JavaScript and machine learning: Google shows what’s possible using the web programming language. From head-controlled Pacman to browser-based upload filters — here are some of the best uses of TensorFlow.js.

Building and training machine-learning models using a web-scripting language might seem ambitious, but in 2019 it’s perfectly feasible.

Helping make machine learning possible in the browser is TensorFlow.js, Google’s open-source library for carrying out machine learning using JavaScript. The possibilities opened up by the library were showcased recently with a Google Doodle that generated a fresh Bach-style melody on demand.

In a talk at the Google I/O conference last week Sandeep Gupta, product manager at Google, said TensorFlow.js could be used by developers to create new machine-learning models, as well as to run or retrain pre-trained models.

“It’s built specifically to make it easier for JavaScript developers to build and use machine-learning models within their JavaScript applications,” he said, adding that machine-learning models built using the broader TensorFlow framework can be converted to run using TensorFlow.js.

TensorFlow.js can be used in JavaScript applications running in the browser, on servers inside a Node.js environment, on the desktop using Electron and on mobile browsers on Android and iOS devices. However, it is within the browser that Gupta sees the most possibilities.

“We see a ton of use cases in the browser and it has a lot of advantages because the browser is super interactive, you have easy access to sensors, such as webcams and microphones, which you can then bring into your machine-learning models,” he said.

“Also we use WebGL-based acceleration, so if you have a GPU in your system you can take advantage of that and get really good performance.”

That said, not every developer is won over by TensorFlow.js, with some arguing the library still has significant limitations.

So what exactly is possible using TensorFlow.js? While the framework is still relatively new, only hitting 1.0 this year, Gupta said there has been “really good adoption and usage by the community”, and used his talk to demo some of the most interesting uses of TensorFlow.js.

What’s possible with machine learning and JavaScript?

WeChat Pacman

You might not be crying out for a new way to play Pacman, but TensorFlow.js has made a novel spin on the classic arcade game possible.

Gupta showed a head-controlled version of the game, a JavaScript application running on the WeChat social media messaging platform on a smartphone.

After a quick calibration step, Gupta was able to control Pacman using head gestures, tracked by his phone’s camera, looking left to move left, right to move right, and so on.

“It’s a really fun way of interacting with the device, and the nice thing is that you can do a variety of things using web cams, using text, using speech, and have a very convenient way of sharing these applications without having to install anything,” he said.

Gupta looks to the left to move Pacman around the maze. Image: Google

Uber Manifold

The taxi and delivery company Uber uses machine learning to tackle a wide variety of problems at a very large scale.

Helping it achieve that is Manifold, a browser-based application that Uber uses to visualize and debug their machine-learning models and data pipelines.

“This application runs in the browser and they’re using TensorFlow.js for a lot of numerical computations, so for example distance calculations and visualization, as well as clustering of data,” said Gupta, adding that “because of the WebGL acceleration, they could accelerate these computations more than 100x compared to just natively using JavaScript”.

AirBnB Identity Document Detection

Online property rental service AirBnB uses machine learning in the browser to stop people from inadvertently uploading sensitive information when adding a picture to their profile.

“When a user is trying to upload a profile picture to the AirBnB website, sometimes people accidentally use a driver’s license picture or a passport picture, which may end up containing personal sensitive information,” said Gupta.

“So AirBnB runs a machine-learning model client-side in the browser or on device, so that if you were to choose a picture which may have such sensitive information it will alert you before you upload that picture.”

Clinic.js

Clinic.js provides a tool for sys admins and software engineers to profile server-side performance in a Node.js environment.

“This is a node.js-based application which is used for profiling node jobs or node processes and they’re using TensorFlow.js to look for anomalies, or spikes in CPU usage or memory consumption of these node applications,” said Gupta.

Google Creatability

One of the main destinations for showing what is possible using TensorFlow.js is Creatability, Google’s creative labs team’s showcase for experiments using machine learning.

Gupta showed a machine-learning powered demo that allows a person to play a piano keyboard using head gestures.

Other examples of using TensorFlow with JavaScript online include Google’s Gallery page for TensorFlow.js and Magenta.js plug-ins offering machine-learning models for music generation.

Competitors to TensorFlow

TensorFlow isn’t alone in the deep learning field; in fact, there are a number of other companies with machine learning frameworks, including the following.

How do I start using TensorFlow?

The first thing any developer should do is read the TensorFlow Getting Started guide, which includes a TensorFlow Core Tutorial. If you’re new to machine learning, make sure to check out the following guides:

Developers can install TensorFlow on Linux, Mac, and Windows (or even install from source), or check out their various tools from the official TensorFlow GitHub page.

Finally, developers can take advantage of all the TensorFlow guides:

JavaScript and machine learning: Google shows what’s possible using the web programming language