Universe npm

Hello friends!

I wanted to share with you my latest mini-project: three-dimensional visualization of all npm. The universe looks like this:


')
You can fly with the WASD keys, or if you watch it from the phone - just rotating the phone around (the phone must support WebGL).

Many packages in the center depend on monsters like lodash, request. Many insulated packages that form a ring of asteroids:



All sources here: github.com/anvaka/allnpmviz3d/
An interactive version is available here: anvaka.imtqy.com/allnpmviz3d/

Video demo:



I hope you will like it! Under the cut there are some technical details about how this is done.

How is this done?

The project is based on angular.js, twitter bootstrap and three.js. All project sources are written in CommonJS style, which makes it easy to structure the program and use ready-made modules from npm. I use gulp to create the final file. In gulpfile , browserify is started to process commonjs packages, and less to build bootstrap styles.

The entire graph initially fits on the server using modules from ngraph and utilities from the allnpm package.

We send the graph to the client

Send a graph with information about 100 thousand nodes and 200+ thousand connections in the forehead (for example, as JSON) turned out to be expensive - the graph is too large. Therefore, he invented a special binary format for compact graph storage.

In this format, each node is defined by three 32-bit integers: the coordinates of the node x, y, z. The position of the nodes is sent in a continuous stream to the client. The client reads triples and remembers their index:

Index Position 1 x1, y1, z1 2 x2, y2, z2 

As soon as the client has received this data, he begins to draw nodes in space. But how to convey the connection? Links are stored in a separate binary file. The file consists of consecutively written int32. Some numbers are negative, some are positive. With negative numbers, I mark the beginning of the next node, and the positive ones tell which indices the last negative is associated with.

For example:

 -1 2 -3 4 5 

This means that vertex 1 is connected with vertex 2, and vertex 3 is connected with vertices 4 and 5. As a result, the entire graph is placed in 2.5 MB.

Finally, a separate json array sends information about all the names. The index in the array corresponds to the index in the file with positions.

Isn't angular slow?

Looking for what. Angular.js is an excellent library for building UIs. I use angular only to display search results and prompts in the interface. The scene itself is drawn completely independent of angular.

Many of my colleagues say that choosing angular you choose all or nothing. It is difficult to refuse / replace angular if you have already started writing on it. I think they are right, because the introduction of dependencies and the system of modules in angular are extremely narrowly targeted.

But, if you stop using angular.module (), the code suddenly stops looking like an angular code. These are the usual javascript functions, with the usual parameters. And if so, they too can be easily created and shared on npm'e.

So I did. Replaced angular.module with a simple package that “accumulates” angular-like functions:

 module.exports = require('an').controller(myController); function myControlelr($scope) { $scope.message = " !"; } 

and then at one stroke registers the accumulated functions when the site is launched:

 var ngApp = angular.module('allnpmviz3d', []); require('an').flush(ngApp); angular.bootstrap(document, [ngApp.name]); //  myController -      angular.js 

Final Notes

Thanks to npm and commonjs, the project consists of approximately 1,700 lines of my code and 63,300 not mine:



I know that measuring code with strings is so-so scientifically, but overall, the perception of the complexity of a project has been greatly improved with a modular approach.

The project also has an Easter egg. Since you, dear reader, have reached the very end, try entering in the search field

: i love npm

and the stars must answer you.

Hope you enjoyed it! I will be glad to your advice and comments!