Node.JS Loading modules on demand

Sometimes, for example, when processing large data arrays, in order to use maximum environmental resources and reduce the total working time spent, we have to use competing processes that simultaneously perform similar tasks on different objects.

Suppose we are developing a simple package for npm . Let's call it, for example, storage . We will foresee the possibility of using one of several types of storage, for example, FsStorage (file storage), MysqlStorage (MySQL storage), MongoStorage (Mongo storage).

Throw the contents of the source code of our package (under the spoiler):


The immediate dependency code of mysql and mongoose is not necessary for the demonstration. Therefore, let's place mysql and mongoose stubs (under the spoiler) instead of the code:


Then the package's file structure will look like this (under the spoiler):


Now imagine that we have a mega-big task that we will perform with the help of several child processes, and in the course of execution we need to use the storage as files.

 var storage = require('storage'); var fsStorage = new storage.FsStorage(); 

We start and we observe: each child process occupies memory 10 times more than expected. And if the number of competing processes exceeds a hundred, and this is not the only task that is performed on the server in real time ?!

Here we understand that, for example, when using file storage, there is no need to load both the MySQL database management library and the Mongo ODM client.

Immediately after calling require('storage') , messages are displayed in the console:

 MySQL module loaded Mongoose module loaded 

Object.defineProperty() experimented with the Object.defineProperty() method, I achieved an amazing result, which I designed as a function demandProperty() :

 function demandProperty(obj, name, modPath) { Object.defineProperty(obj, name, { configurable: true, enumerable: true, get: function() { var mod = require(modPath); Object.defineProperty(obj, name, { configurable: false, enumerable: true, value: mod }); return mod } }) } 

The principle of operation is simple: Instead of a direct link, for example, to MysqlStorage() , an accessor ( getter ) is created. With any request to the accessor, require() triggered, and the accessor itself returns the result of require() . In addition, using the same Object.defineProperty() we set back a direct link to the same require() result instead of the accessor (that is, on MysqlStorage() ). So all requests (of course, except the first) will work with the same speed and reliability from leaks, as if we had left the classic require() .

Change lib/index.js . Replace:

 module.exports = { FsStorage: require('./fs-storage.jsx'), MysqlStorage: require('./mysql-storage.jsx'), MongoStorage: require('./mongo-storage.jsx'), }; 

on:

 demandProperty(module.exports, 'FsStorage', './fs-storage.jsx'); demandProperty(module.exports, 'MysqlStorage', './mysql-storage.jsx'); demandProperty(module.exports, 'MongoStorage', './mongo-storage.jsx'); 

And we use:

 var storage = require('storage'); console.log(util.inspect(storage)); /* => { FsStorage: [Getter], MysqlStorage: [Getter], MongoStorage: [Getter] } */ console.log(util.inspect(storage.FsStorage.name)); // => 'FsStorage' console.log(util.inspect(storage)); /* => { FsStorage: [Function: FsStorage], MysqlStorage: [Getter], MongoStorage: [Getter] } */ var mysql = new storage.MysqlStorage(); // => MySQL module loaded console.log(util.inspect(mysql)); // => '{}' console.log(util.inspect(storage)); /* => { FsStorage: [Function: FsStorage], MysqlStorage: [Function: MysqlStorage], MongoStorage: [Getter] } */ 

There is another subtlety. If the definition of the demandProperty() function is demandProperty() outside the modules in the lib folder, the last argument must be passed the full path to the module, otherwise require() will look for the module in the folder where demandProperty() defined:

 demandProperty(module.exports, 'FsStorage', path.resolve(__dirname, './fs-storage.jsx')); demandProperty(module.exports, 'MysqlStorage', path.resolve(__dirname, './mysql-storage.jsx')); demandProperty(module.exports, 'MongoStorage', path.resolve(__dirname, './mongo-storage.jsx')); 

All successful experiments!