Machine learning with Node.js using the Tensorflow.js library
Hello everyone, colleagues!
Perhaps the fans of the Tensorflow library, who have already noticed this book in pre-order, also looked closely at the possibilities of machine and in-depth training in the browser, especially since the topic was not overlooked by Francois Chollet himself. We invite those interested in the cat section, where it is described how images are recognized using the Tensorflow.js library.
TensorFlow.js is a new version of the popular open source library that enriches the JavaScript language with deep learning capabilities. Developers can now define, train, and run models using the high-level library API .
Thanks to pre-trained models, developers can now easily solve such complex tasks as pattern recognition , music generation, or defining human poses in just a few lines of JavaScript.
')
Tensorflow.js started as a front-end browser library, but this year experimental support for Node.js was added to it. Thus, TensorFlow.js can be used in JavaScript backend applications, which completely eliminates the need to resort to Python.
Reading about this library, I decided to try it on a simple task ...
However, after a few days of chiselling, I did it anyway! Hooray!
Before turning to a detailed analysis of the code, let's talk about other implementations of the TensorFlow library.
Tensorflow
TensorFlow is a free software library for machine learning applications. TensorFlow can be used to create neural networks and implement other deep learning algorithms.
This library, released by Google in November 2015, was originally written in Python . In order to train and evaluate the models created in it, calculations on a CPU or GPU are applied. Initially, this library was created to work on high-performance servers using resource-intensive GPUs.
Recent updates have made it possible to optimize this library and use it in environments with more limited resources — for example, on mobile devices and in web browsers.
TensorFlow Lite
Tensorflow Lite , a lightweight version of this library for mobile and embedded systems, was released in May 2017. Along with it, a new set of pre-trained deep models for problems related to pattern recognition is provided; This collection is called MobileNet . MobileNet models have been designed specifically to work effectively in environments with limited resources, such as mobile devices.
TensorFlow.js
Following Tensorflow Lite in March 2018, TensorFlow.js was announced . This version of the library is designed to work in the browser and is based on an earlier project called deeplearn.js . WebGL provides GPU library access. Developers use the JavaScript API to train, load, and run models.
Later TensorFlow.js was extended to work with Node.js, for this the library add
Import existing models in TensorFlow.js
TensorFlow and Keras ready-made models can be implemented using the TensorFlow.js library. Before running the model, you need to translate into a new format with this tool . Pre-trained and transformed models for image classification, posturing and k-nearest-neighbor detection are available on Github .
Using TensorFlow.js with Node.js
Installing TensorFlow libraries
TensorFlow.js can be installed from the NPM registry .
Both extensions for Node.js use native dependencies that will be compiled on demand.
Loading TensorFlow libraries
The JavaScript API for Tensorflow is provided from the core library. Extension modules that support Node.js do not provide additional APIs.
Loading TensorFlow models
TensorFlow.js provides the NPM library (
The MobileNet model for image classification is a deep neural network, trained to distinguish between 1000 different image classes .
The following code is used as an example in the project's README file for the model loading.
One of the first problems I encountered was that this code does not work with Node.js.
After examining the source code , we see that the mobilenet library is a wrapper for the
The Node.js extension at the time of this writing did not yet support HTTP requests for dynamic model retrieval. It remained only to manually upload the model to the file system.
However, after reading the library source code, I found a workaround ...
Loading models from the file system
If the MobileNet class is created manually, you can not call the module's
Cool, it works!
But where do the model files come from?
MobileNet Models
Models for TensorFlow.js consist of two types of files: model configuration file, stored in JSON format, and model weights, stored in binary format. Model weights are often fragmented into multiple parts to optimize browser caching.
After reviewing the automatic load code for MobileNet models, we see that the models, their configurations and weight fragments are retrieved from the public container at the following address.
The template parameters in the URL describe the model versions listed here . The resulting classification accuracy is also displayed on the same page.
The source code states that only models of the MobileNet v1 version can be downloaded using the
The HTTP retrieval code loads the
Manual download of models
If you need to save all the model files in the file system, you can do this: extract the model configuration file, parse the syntax of all weight files referenced in the configuration file, and then download each weight file manually.
I was going to use the MobileNet V1 module with an alpha value of 1.0 and an image size of 224 pixels . So I get the following URL for the model configuration file.
Once this file is downloaded locally, you can use the
Using the
The
Image classification
This sample code provided with TensorFlow.js demonstrates how to return the result of image classification.
This does not work in Node.js due to the lack of DOM support.
The
I decided not to try to use the external package to simulate the DOM element manually, but found that
We generate Tensor3D from the image
Reading the source code of the method used to convert the DOM elements to the Tensor3D classes, we find that the following input parameters are used to generate the Tensor3D class.
Creating input values for JPEG
The
As a result, we obtain a Uint8Array with four channel values (
Input Requirements for MobileNet Models
The MobileNet model used here classifies images with a height and width of 224 pixels. Input tensors must contain floating-point values in the range from -1 to 1 for each of the three channel values of each pixel.
Input values for images with a different dimension must be recalculated to the correct size before classification. In addition, pixel values obtained from a JPEG decoder are in the range of 0 - 255, and not -1 - 1. These values also need to be converted before classification.
TensorFlow.js has library methods that simplify this process, but, even better, there is a special library
The developer can transfer input Tensor3D of type
Getting forecasts
MobileNet models in Tensorflow learn how to recognize objects from 1000 of the most important classes from the ImageNet data set . At the output of the model, probabilistic values are given, which characterize the chances of finding these objects in the classified image.
A complete list of the trained classes for the model used is in this file .
The
Example
So, we figured out how to use the TensorFlow.js library and MobileNet models in Node.js, and now we will look at how this script classifies the image specified as a command line argument.
Source
Save this script file and package descriptor in local files.
Testing
Download the model files to the mobilenet directory, following the instructions above.
Install project dependencies with NPM
Download a sample JPEG file for classification
Run the script, whose arguments will be the model file and the input image.
If everything worked correctly, then the following output should appear in the console.
The image is correctly classified as containing a panda with a probability of 99.93%!
Conclusion
The TensorFlow.js library opens up deep learning opportunities for JavaScript developers. The use of pre-trained models with the TensorFlow.js library makes it easy to build new applications in JavaScript applications to solve complex machine learning tasks, with minimal effort and concise code.
The TensorFlow.js library was created especially for working in the browser, but now it interacts with Node.js, although not all tools and utilities support this new execution environment. Having fiddled with the library for several days, I learned to use it with MobileNet models for visual recognition of images from a local file.
Perhaps the fans of the Tensorflow library, who have already noticed this book in pre-order, also looked closely at the possibilities of machine and in-depth training in the browser, especially since the topic was not overlooked by Francois Chollet himself. We invite those interested in the cat section, where it is described how images are recognized using the Tensorflow.js library.
TensorFlow.js is a new version of the popular open source library that enriches the JavaScript language with deep learning capabilities. Developers can now define, train, and run models using the high-level library API .
Thanks to pre-trained models, developers can now easily solve such complex tasks as pattern recognition , music generation, or defining human poses in just a few lines of JavaScript.
')
Tensorflow.js started as a front-end browser library, but this year experimental support for Node.js was added to it. Thus, TensorFlow.js can be used in JavaScript backend applications, which completely eliminates the need to resort to Python.
Reading about this library, I decided to try it on a simple task ...
Use TensorFlow.js to visually recognize patterns on images when using JavaScript from Node.jsUnfortunately, the documentation and code samples basically describe the use of this library in the browser. Project utilities designed to simplify the loading and use of pre-trained models at the time of this writing did not yet support Node.js. I had to spend a lot of time to thoroughly read the Typescript sources for this library.
However, after a few days of chiselling, I did it anyway! Hooray!
Before turning to a detailed analysis of the code, let's talk about other implementations of the TensorFlow library.
Tensorflow
TensorFlow is a free software library for machine learning applications. TensorFlow can be used to create neural networks and implement other deep learning algorithms.
This library, released by Google in November 2015, was originally written in Python . In order to train and evaluate the models created in it, calculations on a CPU or GPU are applied. Initially, this library was created to work on high-performance servers using resource-intensive GPUs.
Recent updates have made it possible to optimize this library and use it in environments with more limited resources — for example, on mobile devices and in web browsers.
TensorFlow Lite
Tensorflow Lite , a lightweight version of this library for mobile and embedded systems, was released in May 2017. Along with it, a new set of pre-trained deep models for problems related to pattern recognition is provided; This collection is called MobileNet . MobileNet models have been designed specifically to work effectively in environments with limited resources, such as mobile devices.
TensorFlow.js
Following Tensorflow Lite in March 2018, TensorFlow.js was announced . This version of the library is designed to work in the browser and is based on an earlier project called deeplearn.js . WebGL provides GPU library access. Developers use the JavaScript API to train, load, and run models.
Later TensorFlow.js was extended to work with Node.js, for this the library add
tfjs-node on tfjs-node .Import existing models in TensorFlow.js
TensorFlow and Keras ready-made models can be implemented using the TensorFlow.js library. Before running the model, you need to translate into a new format with this tool . Pre-trained and transformed models for image classification, posturing and k-nearest-neighbor detection are available on Github .
Using TensorFlow.js with Node.js
Installing TensorFlow libraries
TensorFlow.js can be installed from the NPM registry .
@tensorflow/tfjs- Core TensorFlow.js Library@tensorflow/tfjs-node- TensorFlow.js Node.js extension@tensorflow/tfjs-node-gpu- TensorFlow.js Node.js extension with support for GPU computing
npm install @tensorflow/tfjs @tensorflow/tfjs-node // ... npm install @tensorflow/tfjs @tensorflow/tfjs-node-gpu Both extensions for Node.js use native dependencies that will be compiled on demand.
Loading TensorFlow libraries
The JavaScript API for Tensorflow is provided from the core library. Extension modules that support Node.js do not provide additional APIs.
const tf = require('@tensorflow/tfjs') // ( CPU) require('@tensorflow/tfjs-node') // ( GPU) require('@tensorflow/tfjs-node-gpu') Loading TensorFlow models
TensorFlow.js provides the NPM library (
tfjs-models ), which tfjs-models easy to load pre-trained and transformed models for classifying images , determining poses, and finding the k-nearest neighbors .The MobileNet model for image classification is a deep neural network, trained to distinguish between 1000 different image classes .
The following code is used as an example in the project's README file for the model loading.
import * as mobilenet from '@tensorflow-models/mobilenet'; // const model = await mobilenet.load(); One of the first problems I encountered was that this code does not work with Node.js.
Error: browserHTTPRequest is not supported outside the web browser. After examining the source code , we see that the mobilenet library is a wrapper for the
tf.Model class. When called, the load() method automatically loads the necessary model files located at the external HTTP address and instantiates the TensorFlow model.The Node.js extension at the time of this writing did not yet support HTTP requests for dynamic model retrieval. It remained only to manually upload the model to the file system.
However, after reading the library source code, I found a workaround ...
Loading models from the file system
If the MobileNet class is created manually, you can not call the module's
load method, but overwrite the automatically generated path variable containing the model's HTTP address by replacing this address with the local path in the file system. After that, when calling the load method in an instance of a class, the file system loader class will work; in this case, we refuse to use the HTTP browser loader. const path = "mobilenet/model.json" const mn = new mobilenet.MobileNet(1, 1); mn.path = `file://${path}` await mn.load() Cool, it works!
But where do the model files come from?
MobileNet Models
Models for TensorFlow.js consist of two types of files: model configuration file, stored in JSON format, and model weights, stored in binary format. Model weights are often fragmented into multiple parts to optimize browser caching.
After reviewing the automatic load code for MobileNet models, we see that the models, their configurations and weight fragments are retrieved from the public container at the following address.
https://storage.googleapis.com/tfjs-models/tfjs/mobilenet_v${version}_${alpha}_${size}/ The template parameters in the URL describe the model versions listed here . The resulting classification accuracy is also displayed on the same page.
The source code states that only models of the MobileNet v1 version can be downloaded using the
tensorflow-models/mobilenet .The HTTP retrieval code loads the
model.json file from the storage location, and then recursively selects all model fragments with weights referenced. These are files in the format groupX-shard1of1 .Manual download of models
If you need to save all the model files in the file system, you can do this: extract the model configuration file, parse the syntax of all weight files referenced in the configuration file, and then download each weight file manually.
I was going to use the MobileNet V1 module with an alpha value of 1.0 and an image size of 224 pixels . So I get the following URL for the model configuration file.
https://storage.googleapis.com/tfjs-models/tfjs/mobilenet_v1_1.0_224/model.json Once this file is downloaded locally, you can use the
jq tool to parse the names of all weight files. $ cat model.json | jq -r ".weightsManifest[].paths[0]" group1-shard1of1 group2-shard1of1 group3-shard1of1 ... Using the
sed tool, you can put in front of the name of each HTTP URL element to generate a URL for each weight file. $ cat model.json | jq -r ".weightsManifest[].paths[0]" | sed 's/^/https:\/\/storage.googleapis.com\/tfjs-models\/tfjs\/mobilenet_v1_1.0_224\//' https://storage.googleapis.com/tfjs-models/tfjs/mobilenet_v1_1.0_224/group1-shard1of1 https://storage.googleapis.com/tfjs-models/tfjs/mobilenet_v1_1.0_224/group2-shard1of1 https://storage.googleapis.com/tfjs-models/tfjs/mobilenet_v1_1.0_224/group3-shard1of1 ... The
parallel and curl commands allow you to then download all these files to my local directory. cat model.json | jq -r ".weightsManifest[].paths[0]" | sed 's/^/https:\/\/storage.googleapis.com\/tfjs-models\/tfjs\/mobilenet_v1_1.0_224\//' | parallel curl -O Image classification
This sample code provided with TensorFlow.js demonstrates how to return the result of image classification.
const img = document.getElementById('img'); // const predictions = await model.classify(img); This does not work in Node.js due to the lack of DOM support.
The
classify method accepts various DOM elements ( canvas , video , image ) and automatically extracts and converts the “picture” bytes from these elements into the tf.Tensor3D class used as input to the model. Alternatively, the tf.Tensor3D input information can be transmitted directly.I decided not to try to use the external package to simulate the DOM element manually, but found that
tf.Tensor3D easier to manually assemble tf.Tensor3D .We generate Tensor3D from the image
Reading the source code of the method used to convert the DOM elements to the Tensor3D classes, we find that the following input parameters are used to generate the Tensor3D class.
const values = new Int32Array(image.height * image.width * numChannels); // , const outShape = [image.height, image.width, numChannels]; const input = tf.tensor3d(values, outShape, 'int32'); pixels is a two-dimensional array of type (Int32Array) containing a sequential list of channel values for each pixel. numChannels is the number of channel values per pixel.Creating input values for JPEG
The
jpeg-js library is a JPEG encoder / decoder for Node.js, written in pure JavaScript. Using this library, you can extract RGB values for each pixel. const pixels = jpeg.decode(buffer, true); As a result, we obtain a Uint8Array with four channel values (
RGBA ) per pixel ( width * height ). In the MobileNet model, only three color channels ( RGB ) are used for classification, the alpha channel is ignored. This code converts a four-channel array into the correct three-channel version. const numChannels = 3; const numPixels = image.width * image.height; const values = new Int32Array(numPixels * numChannels); for (let i = 0; i < numPixels; i++) { for (let channel = 0; channel < numChannels; ++channel) { values[i * numChannels + channel] = pixels[i * 4 + channel]; } } Input Requirements for MobileNet Models
The MobileNet model used here classifies images with a height and width of 224 pixels. Input tensors must contain floating-point values in the range from -1 to 1 for each of the three channel values of each pixel.
Input values for images with a different dimension must be recalculated to the correct size before classification. In addition, pixel values obtained from a JPEG decoder are in the range of 0 - 255, and not -1 - 1. These values also need to be converted before classification.
TensorFlow.js has library methods that simplify this process, but, even better, there is a special library
tfjs-models/mobilenet that automatically solves this problem !The developer can transfer input Tensor3D of type
int32 , as well as various dimensions to the classify method, which prior to classification translates the input values into the correct format. That is, we do not have to do anything here. Super!Getting forecasts
MobileNet models in Tensorflow learn how to recognize objects from 1000 of the most important classes from the ImageNet data set . At the output of the model, probabilistic values are given, which characterize the chances of finding these objects in the classified image.
A complete list of the trained classes for the model used is in this file .
The
tfjs-models/mobilenet offers a classify method in the MobileNet class, which returns the top X most likely classes, based on what is shown in the picture. const predictions = await mn_model.classify(input, 10); predictions is an array of X classes and probabilities in the following format. { className: 'panda', probability: 0.9993536472320557 } Example
So, we figured out how to use the TensorFlow.js library and MobileNet models in Node.js, and now we will look at how this script classifies the image specified as a command line argument.
Source
Save this script file and package descriptor in local files.
{ "name": "tf-js", "version": "1.0.0", "main": "script.js", "license": "MIT", "dependencies": { "@tensorflow-models/mobilenet": "^0.2.2", "@tensorflow/tfjs": "^0.12.3", "@tensorflow/tfjs-node": "^0.1.9", "jpeg-js": "^0.3.4" } } const tf = require('@tensorflow/tfjs') const mobilenet = require('@tensorflow-models/mobilenet'); require('@tensorflow/tfjs-node') const fs = require('fs'); const jpeg = require('jpeg-js'); const NUMBER_OF_CHANNELS = 3 const readImage = path => { const buf = fs.readFileSync(path) const pixels = jpeg.decode(buf, true) return pixels } const imageByteArray = (image, numChannels) => { const pixels = image.data const numPixels = image.width * image.height; const values = new Int32Array(numPixels * numChannels); for (let i = 0; i < numPixels; i++) { for (let channel = 0; channel < numChannels; ++channel) { values[i * numChannels + channel] = pixels[i * 4 + channel]; } } return values } const imageToInput = (image, numChannels) => { const values = imageByteArray(image, numChannels) const outShape = [image.height, image.width, numChannels]; const input = tf.tensor3d(values, outShape, 'int32'); return input } const loadModel = async path => { const mn = new mobilenet.MobileNet(1, 1); mn.path = `file://${path}` await mn.load() return mn } const classify = async (model, path) => { const image = readImage(path) const input = imageToInput(image, NUMBER_OF_CHANNELS) const mn_model = await loadModel(model) const predictions = await mn_model.classify(input) console.log('classification results:', predictions) } if (process.argv.length !== 4) throw new Error('incorrect arguments: node script.js <MODEL> <IMAGE_FILE>') classify(process.argv[2], process.argv[3]) Testing
Download the model files to the mobilenet directory, following the instructions above.
Install project dependencies with NPM
npm installDownload a sample JPEG file for classification
wget http://bit.ly/2JYSal9 -O panda.jpgRun the script, whose arguments will be the model file and the input image.
node script.js mobilenet/model.json panda.jpgIf everything worked correctly, then the following output should appear in the console.
classification results: [ { className: 'giant panda, panda, panda bear, coon bear', probability: 0.9993536472320557 } ] The image is correctly classified as containing a panda with a probability of 99.93%!
Conclusion
The TensorFlow.js library opens up deep learning opportunities for JavaScript developers. The use of pre-trained models with the TensorFlow.js library makes it easy to build new applications in JavaScript applications to solve complex machine learning tasks, with minimal effort and concise code.
The TensorFlow.js library was created especially for working in the browser, but now it interacts with Node.js, although not all tools and utilities support this new execution environment. Having fiddled with the library for several days, I learned to use it with MobileNet models for visual recognition of images from a local file.
Source: https://habr.com/ru/post/432984/
All Articles