Server Visualization: NodeJS + D3.js + PhantomJS

We had a whim on a project - to draw graphics on the server side, and not simple ones, but as close as possible to the existing graphics on the client side .
Yes, exactly, on the client there were already all sorts of beautiful, implemented on d3.js.
To investigate the possibilities, a comprehensive analysis method “google-driven investigation” was applied and in the first iteration the choice fell on the node + phantom .

For details, I ask in the depths of the post.

Boring introduction

I will tell in brief about the project to describe the situation. Our company has found a BigData start-up, the team has won a tender and now the four of us are sawing analytics in the cloud for heavy-weight datasets.
Our zoo consists of clusters on AWS with auto-warming, Scala , Spark , Shark , Mesos , NodeJS and other scary technologies (I hope this project will allow me and my colleagues to satisfy intellectual hunger and write a couple of articles).
')

Disclaimer

Our team is two experienced javista and two "polyglots" (java / scala + javascript). We consider ourselves as good engineers and use languages as tools, although we are focusing on java. Therefore, if the material seems “non-Orthodox” from the point of view of approaches and practices, I ask you to throw rotten eggs in a personal, and constructive criticism in the commentary.

We have weekly iterations and retrospective + demo at the end of the week. This imposes a number of restrictions on research and the search for best practices.
At the time of the implementation of the solution, we already had “digital camera” on a cliff and rest-services on the node.

The essence

Requirements

Graphics must be static
The graphics should be as similar to the client interactive "whistles"
Graphics must be generated on the server.
Interface interaction - REST
All business should be under construction dynamically on datasets from storage

Why noda and phantom?

During a cursory study of the problem, three options were discovered:

Use the js implementation of the house tree and Image Magic to convert SVG to PNG (an example was found ).
Use java libraries for charts in the rock (or rock analogs ) and maximally stylize them under d3
Use phantom in conjunction with rock / node

Option number 1 left open the question of css-styles and general expediency (not nodovskoe is the vocation of the processor to load calculations).
Option number 2 seemed reasonable, but guaranteeing lasting pain in the sciatic nerve.
It was decided to use Option number 3.

Subsequent studies and experiments have shown that:

Scala is not friendly with phantom. And no phantom provides external api.
But the phantom is friends with the node. And there are several npm-modules that provide a bridge between the node and the native api phantom.

What kind of bridge?

It turned out to be interesting. Due to the fact that a stale module was originally chosen to work with a phantom, I had to dive headlong into the module debugging and trolling the community on the githaba to support self-written modules.

It turned out that the phantom has no external api at all. Even for the node. But internal api is emulated through socket.io and by redefining the alert handler on the page opened in the phantom.

Author uvuhauha for resourcefulness!

Algorithm like this:

A script is created that will receive socket.io messages inside the phantom
A stub page is created with a script attached.
Override the alert listener that will contain the “response” page on the socket.io message
At the node, an express server rises up, rendering the page and processing socket.io requests.
The phantom process starts and is fed to a stub page.
The module exports “mirrored” api phantom (but all methods become asynchronous; in phantom, they are almost all synchronous)

Having gone deep into the “phantom + node” option, I found out that it is possible to use the already existing client javascript code for plotting on the server side.
Phantom is a webcam with a full-fledged implementation of a tree-house, styles and javascript. And it allows you to take pictures of the rendered page. Such a solution allows not to duplicate the graphics construction code at all!

Underwater rocks

For the phantom to work, it must be installed in the system :)
sudo apt-get install phantomjs
or
brew install phantomjs
After these magic words, the bridge will be able to use the webpage module.

During the implementation I had to sweat with the use of a phantom through the node. The first module turned out to be rather bad and crooked (see previous spoiler), because the choice fell on node-phantom .
There was a long-standing problem like the world - the lack of documentation on api.

By the method of scientific spear it was found out that:

Phantom injectit ( page.indectJs ) scripts into a page only along the full path on the file system.
Phantom inclusive ( page.includeJs ) scripts per page for the full URL, but in the module the page API of the internal API page.includeJs is corrupted due to implementation features.
Due to the position of the stars in the sky, the phantom does not parse styles that are dynamically connected by adding ..
Parameters passed for processing into the phantom page must be serialized to a string.

The long-awaited decision

I use the vow vow module to reduce the macaroni code. Bad or good use - write in the comments!

 //       (    package.json) var phantom = require("node-phantom") //  , vow = require("vow") //   - , cfg = require("../config") //       , fs = require('fs') //      , pi; //          exports.init = function () { if (pi) { pi.exit(); } phantom.create(function (err, instance) { pi = instance; }); } //        -   exports.render = function (dataset, opts) { var promise = vow.promise(); //       pi.createPage(function (err, page) { //       ,   page.set("viewportSize", opts.viewport); //    d3    (.  " ") var d3Path = __dirname + "/../client/scripts/vendor/d3.v3.js"; //     ,    d3 // type -    (line, bar, pie) //   chart.xxx.js      var chartJs = __dirname + "/../client/scripts/chart." + opts.type + ".js"; //        var chartCss = __dirname + "/../client/styles/charts.css"; var innerStyle = ""; //   //    ? ?    injectLib_(page, d3Path)() .then(injectLib_(page, chartJs)) .then(readCssStyles_(chartCss)) .then(drawChart_(page, {dataset: dataset, innerCss: innerStyle}, opts)) .then(function (res) { //   ,       promise.fulfill({filename: res.filename}); }) .fail(function (err) { promise.reject(err) } ) }); return promise; } //       () //   -    " " function readCssStyles_(chartCss) { return function(){ var prom = vow.promise(); fs.readFile(chartCss, 'utf8', function (err,innerCss) { if (err) { console.log(chartCss + ": read failed, err: " + err); prom.reject(chartCss + ": read failed, err: " + err); } else { console.log(chartCss + " read"); prom.fulfill(innerCss); } }); return prom; } } function injectLib_(page, path) { return function () { var prom = vow.promise(); //      ,       page.evaluate page.injectJs(path, function (err) { if (err) { console.log(path + " injection failed") prom.reject(path + " injection failed"); } else { console.log(path + " injected") prom.fulfill(); } }); return prom; } } function drawChart_(page, data, opts) { return function (innerCss) { data.innerCss = innerCss; var prom = vow.promise(); //          //   -  "".        //  ,      ,    page.evaluate(function (data) { //        //    data = JSON.parse(data); //       //     charts.xxx.js charts.line("body",data.dataset); //    ,          var style = document.createElement("style"); style.innerHTML = data.innerCss; document.getElementsByTagName("head")[0].appendChild(style); } , function (err, result) { if (err) { prom.reject("phantomjs evaluation failed : " + err) } //             //   png, pdf, gif  jpeg var filename = cfg.server.chartsPath + '/' + opts.type + "_" + Date.now() + ".png"; var savingPath = "client" + filename; //        page.render(savingPath, function (err, res) { console.log("Saving image: " + filename); page.close(); prom.fulfill({filename: filename}); }); }, JSON.stringify(data)); return prom; } }

Questions, wishes, constructive and trolling - in the comments.
Mistakes in the “great and mighty? - in lichku.
I will be glad to hear your feedback on all aspects - the quality of the code, the quality of the article, the style of presentation.

Source: https://habr.com/ru/post/180927/

All Articles