Welcome, Habrovchane and Skorokodery! In the article about the development of Scorocode, we asked the community what new functionality you would like to see in the service. One of the popular options was support in the server code of npm modules. You asked - we did! For details, we ask under the cat.

Briefly about implementation

To implement the npm modules, we had to completely revise the file and script storage device. We refused to store scripts in a third-party cloud and transferred all data to our servers. This allowed us to create a reserve for the implementation of the necessary functionality, as well as to improve the speed of working with data.

Next came the turn of the JS file handler itself. This handler (hereinafter referred to as “broker”) listens to RabbitMQ and, when a task arrives, sends it to the pool of workers for further processing. The broker itself is written in Go and uses the Go-V8 bridge to execute JavaScript code. It was clear that to support npm modules pure V8 will not work, implementing the most popular and necessary modules on request is also a utopian idea. In any case, the code had to be executed in NodeJS.
')
Many of our competitors have npm modules either unsupported or supported from the whitelist. We immediately determined for ourselves that we want to give our users full access to npm, without limiting them to any list of supported modules. This made us find the ability to run handlers in isolation from the rest, so that the code could not break other people's data.

To solve the problem of running JavaScript code in isolation, we chose Docker. This allowed us to start processing JavaScript code in the container, and not to restrict users in the choice of modules in the future. The container is created from the image when the script runs. The script itself and the installed modules are forwarded in the form of a file and folders inside the container. Additionally, we modified the native require function so that it can support user scripts, installed npm modules, as well as native NodeJS modules.

As we left the admin and developer without beer

This implementation was ready for a long time, we tested it for about a week and, at first glance, everything worked without problems. In an effort to please users as early as possible, we decided to roll out updates on Friday afternoon. After laying out the release, as well as data migration, everything worked with a bang, until our developer tried to write code using setInterval.

As it turned out, when using the timeout parameter of the native vm module, the sandbox runs exactly as much as needed, but as soon as setInterval occurs in our code, timeout does not work. Unfortunately, we did not have time to deal with the problem, and after some deliberation, the task of controlling the execution time of the scripts was placed on the broker’s shoulders.

As a result, the release was laid out, problems were fixed, but the weekend for our administrator and developer began only on Saturday at 1:58, so we had to forget about the glass of Friday beer in the bar. Break the commandment "do not lay out releases on Friday" will not :)

How to use npm modules in Scorocode

In order to connect the necessary modules, you need to go to the “Application Settings” section and select the “npm dependencies” tab.



In the dependencies section, describe all the modules you intend to use. The property is the name of the module, the value is the required version (you can leave the version value empty, in which case the latest version of the module will be installed).

{ "dependencies":{ "scorocode": "", "underscore": "", "async": "" } } 

After that we press the save button and go to the “Server Code” section and create a new script:



As you can see from the screenshot, using installed modules is no different from using NodeJS. To connect the required modules, use require.

 var SC = require('scorocode'); var async = require('async'); var _ = require('underscore'); //  ..... 

Conclusion

In general, we managed to implement the support for npm modules in user applications fairly painlessly. And now we are interested in receiving feedback, how will you use this functionality in your applications?

Also, given the limitations of npm, we are eyeing the support of the package manager for node.js from facebook yarn . Has anyone tried it, what opinions do you have about it?