Impress Application Server in simple terms

This is not the first introductory article about Impress on Habré, but over the past year I have received many questions and gained some experience in explaining the architecture and philosophy of this application server and, I hope, began to better understand the problems and tasks of developers starting its development. Yes, and in the server itself, there have been enough changes for the relevance of a completely new introductory article.

Impress Application Server (IAS) is an application server for Node.js with alternative architecture and philosophy, unlike the mainstream development under the node and designed to simplify and automate a wide range of repeatable tasks, raise the level of application code abstraction, define the scope and structure of applications, optimize both code performance and developer productivity. IAS currently covers only server tasks, but it does it comprehensively, for example, you can combine API, web sockets, streaming, statics, Server-Sent Events on a single port, proxying and URL-rewriting, serve several domains and several applications, as on one server, and on a group of servers working in conjunction, as a whole, as one application server.

Introduction

First, I want to list a number of problems in the conventional approach for node.js, which prompted me to start developing an application server:

1. Application code is often mixed with system code. The point is that the node, and the majority of the framework frameworks, are too low-level and each application necessarily contains a part of the system code that is not related to the tasks of the subject area. So it happens, for example, that adding an HTTP header becomes a method of the Patient class and is in the same file with the task of routing URLs to this patient and sending events via web sockets. This is monstrous.
2. Noda gives excessive freedom in terms of application architecture , which is difficult to digest at once, not only to a beginner, but even to an experienced specialist. In addition to the concept of middleware, which, you see, is not enough for a full development, there are no common architectural patterns. Dividing a project into files, dividing logic into classes, applying patterns, creating internal APIs in applications, isolating layers, and even the directory structure are all left to the developer. As a result, the structure and architecture of projects is very different for different teams and specialists, which complicates the understanding and interconnection of the code.
3. In the world of the node, there is fanatical worship of REST and, as a result, the refusal to store state in the server's memory. This is despite the fact that Node.js applications live in memory for a long time (i.e. they are not loaded with each request and are not completed between requests). Poor use of memory ignoring the possibility of deploying a model of a problem to be solved there for a long time (so that I / O could be reduced to the very minimum) is a crime against performance.
4. A large number of modules in npm are garbage modules , and among the few good ones, it is not easy to find the right one (the number of downloads and stars does not always adequately reflect the quality of the code and the speed of troubleshooting). It is even more difficult to make an integral application from a set of good, even very good modules. Together, they can be unstable and unpredictable. Modules are not sufficiently shielded from each other to eliminate integration conflicts (for example, a module can override res.end or send http headers, while others do not expect this behavior).

There are still many minor problems, and deep grief with catching errors in Node.js is a topic for three volumes filled with tears, blood and coffee (tea). As a consequence of all of the above, the node still raises concerns and, in most cases, is used as an additional tool in conjunction with other server technologies, performing auxiliary work, such as: scripting client application builds, prototyping, or ensuring the delivery of notifications on web sockets. It is very rare to find a large project that has a server part exclusively on the node.

Formulation of the problem

In addition to negative motivation (listed problems), there were also positive driving factors for developing IAS (ideas and objectives):

1. Scaling of node.js applications by more than one server, each of which has its own cluster (cluster of processes interconnected by IPC).
2. Maintain multiple applications in a single process, a cluster of processes or a server farm with a cluster of processes on each.
3. Automatic replacement of the code in memory if it has changed on the disk, even without restarting the application, through monitoring the file system. As soon as the files loaded by the application change, the IAS reads them into memory. At some point there may be several versions of the code in memory, the old one is unloaded as soon as all requests that came before the change are processed, and the new one is already used for the following requests.
4. Synchronization of data structures in memory between processes. Of course, not all memory structures, but only a global fragment of the domain model unfolded in it. Additive changes and transactions are supported, i.e. if some parameter is incremented in parallel in different processes, then these changes merge, because their order is not important.

Impress philosophy

1. Maximum memory usage . Faster asynchronous I / O is only when there is no I / O at all, or it is reduced to a minimum and is performed in lazy mode, and not during requests.
2. Monolithic architecture and high connectivity code, all major modules are integrated, consistent and optimized to work together. Due to this, there are no unnecessary checks, and the behavior when solving typical tasks is always predictable.
3. Multiplexer of ports, hosts, IP, protocols, servers, processes, applications, handlers and methods. Thus, you can combine statics, API, web sockets, SSE, video streaming and large files on one port, process several applications on different domains or multi-domain sites, etc.
4. The principle of an application virtual machine isolated from the environment using sandboxes (sandboxes). Each application has its own context (scope), in which its own libraries and data are loaded. The application provides architectural places for various processors: initialization and finalization, data models and database structure, configuration and installation (first start), updates, migrations, etc.
5. Separation of application and system code. In general, separating layers of abstractions (higher and lower levels) in an application is much more important than sharing logic with a model and a view, within one layer of abstractions (they are sometimes even more efficient to mix).
6. Map the URL to the file system with inheritance and redefinition of the directory tree. But with the ability to programmatically add handlers directly to memory and prescribe routing by hand in addition to automatic routing by directory structure.
7. The brevity of the code (see examples below) is achieved thanks to the advanced built-in API, which assumes everything that is necessary in the overwhelming majority of cases, can be expanded and reused from project to project. Also, the special style of working with visibility zones and splitting the code into files with logical parts of a convenient size contribute to brevity.

Application area

IAS is designed to create several types of applications:

1. Single-page web applications with API and dynamic change of pages on the client without rebooting from the server.
2. Multi-page web applications with a certain degree of dynamics on the pages through the API (the logic is divided into client and server).
3. Multi-page applications with reloading pages for each event (all the logic on the server).
4. Applications with two-way data exchange or stream of events from the server, interactive applications (usually this is an add-on over options 1 and 2).
5. Network API to access the server for native mobile and window applications.

Multiple API creation methods supported

• RPC API - when the URL identifies the network method with a set of parameters, the order of the call is important and the state between the calls is stored both on the client and on the server;
• REST API, when a URL identifies a resource, a limited number of operations can be performed on a resource (for example, HTTP verbs or CRUD), atomic requests, there is no difference in the call order, and there is no state between calls;
• Event bus: a one-way or two-way client-server interaction stream via WebSockets or SSE (Server-Sent Events) used to notify or synchronize the state of objects between the client and the server;
• Or a mixed way.

Handlers

An analogue of middleware for IAS is a handler (handler) - this is an asynchronous function that has two parameters (client and callback) and is located in a separate file (from which it is exported). When a callback is invoked, the IAS application server will know that the processing has ended. If the function does not cause a callback longer than the timeout, then IAS returns HTTP status 408 (Request timeout) and fixes the problem in the logs. If an exception occurs when the handler is called, then IAS takes the answer to the client, catching errors and restoring work in the optimal way, up to deleting and re-creating a sandbox with corrupted or leaked data structures.
')
An example API handler:

module.exports = function(client, callback) { dbAlias.equipment.find({ type: client.fields.type }).toArray(function(err, nodes) { if (!err) callback(nodes); else callback({ message: 'Equipment of given type not found' }, 404); }); } 

Each HTTP request can cause execution of several handlers. For example, if the URL is requested domain.com/api/example/method.json domain.com/api/example/method.json , and IAS is set to /impress , then the execution will start from the /impress/appplications/domain.com/app/api/example/method.json/ directory and goes through the following steps:

• access rights are checked for the access.js file from this directory, the session (if any) and the user account (if there is an session- associated one),
• the request.js handler from this directory is executed (if found), it is executed when any HTTP method is called (get, post ...),
• one of the handlers corresponding to the HTTP request method is executed, for example get.js , put.js , post.js , etc. (if found)
• the end.js handler is executed (if found), it will be called with any HTTP method,
• the response data will be serialized or the page will be patterned (if it is provided by the type of the returned response)
• the result of the request is sent to the client,
• after that, when the client received a response and we do not delay it, the lazy.js handler (if found) is executed, which can, for example, make pending operations, change / recalculate or save data to the database,
• at any stage of execution in the application, an error may occur that causes an unhandled exception, but we do not need to wrap the code in try / catch or create a domain, this is already done in IAS, if an error occurs, the error.js handler will be called (if found).

If the requested directory does not have the required handler, IAS will search for it one directory higher until it reaches /app . If the handler is in the folder, then it can programmatically call the handler from the directory above (or closest up the tree) via client.inherited() . Thus, you can use a directory tree to form inheritance and override handlers. For example, you can generate the response data in the handler /api/example/request.js , and output it in three formats: /api/example/method.json , /api/example/method.html (also contains templates for output to html), /api/example/method.csv (may contain additional actions, for example, generating a table header). Or make a generic error handler for the entire API in the /api/error.js file. This approach gives more flexibility and reduces the size of the code, however, we pay for it with known limitations.

Extensions to directories mean automatic delivery of content of a certain type from them, which means installing certain HTTP headers and converting the result to the desired data format. All this can be overridden manually, but using extensions reduces the amount of code. The following extensions are supported out of the box: .json, .jsonp, .xml, .ajax, .csv, .ws, .sse and this list is simply extended with the help of plug-ins.

Namespaces

Inside the handler you can see the following names, through which we can access the IAS functions and the connected libraries:

• client - an object containing parsed request fields, links to the original request and response, respectively in client.req and client.res , API for handling the request, links to the session, and an authorized user;
• application - an object responsible for the application and containing its configuration, parameters, and the corresponding application server API;
• db is a namespace containing references to all loaded DBMS drivers and established connections to databases; you can access them through db [alias] or db.alias;
• api is a namespace containing references to all embedded and external libraries that have been resolved from the application configuration. For example, api.fs.readFile(...) or api.async.parallel(...);
• api.impress - link to the application server API;
• system global identifiers common to JavaScript and Node.js: require, console, Buffer, process, setTimeout, clearTimeout, setInterval, clearInterval, setImmediate, clearImmediate . But we can, in the configuration, prohibit the use of some of them, for example, by disabling the require application, and giving it only a certain set of libraries automatically loaded into its namespace api.

There is no need to do require in handlers, it is enough to install the libraries in the / impress folder through npm install and connect them via the / config / sandbox.js configuration (first in the IAS config, and then locally in the app config). Further, the libraries are visible in the handlers via api.libName , the built-in libraries are also visible, for example, api.path.extname(...) , etc.

All databases and DBMS drivers are visible through db.name. Connections are configured in /config/databases.js (for each application separately), are established at startup and are automatically restored when communication is lost. Included are drivers for MongoDB, PostgreSQL and MySQL, wrapped in plug-ins for IAS, if you wish, in 30 minutes you can wrap driver plug-ins into any DBMS.

For the html content type, a simple built-in template engine is used, it is needed rather not for the full generation of pages on the server side, but for assembling the layout (the main layout and layout of the interface pieces), as well as for substituting a few values ​​from data structures into html. The template engine contains inlays and iterators, but more complex template management needs to be implemented in the browser using React, Angular, EJS, etc., requesting templates and data separately and collecting them in the browser (using templates), which is typical for dynamic web sites. applications. The built-in templating engine begins rendering from the html.template file and inserts data from client.context.data into it. The @fieldName@ construct @fieldName@ substitute the value from the field, the @[file]@ construct will insert the file.template file, and the @[name]@ ... @[/name]@ construct implements a hash or array iterator with the name name.

For handlers that return serialized data (.json, .jsonp, .csv, etc.), template matching is not needed. For them, the data structure client.context.data simply serialized to JSON (with recursion clipping). For convenience, you can return the data structure from the handler by the first parameter callback({ field: "value" }); If one handler returned data to the callback or assigned it to client.context.data , then the following ones (until the end of the life of the current HTTP request) can read and modify the data.

Handlers can change the http status code, add their http headers, but in normal mode they only work with a client object that has secure API methods: client.error(code), client.download(filePath, attachmentName, callback), client.cache(timeout), client.end(output) , etc. Starting from version 0.1.157, IAS provides partial support for middleware handlers with 3 parameters: req, res, and next. But this is extremely rare, and the code ported from projects to express or connect can usually be rewritten several times shorter and simpler.

Create handlers of both types, i.e. handler (with 2 parameters) and middleware (with 3 parameters) can be not only from files, but adding routing manually, through method calls, for example:

 application.get('/helloWorld.ajax', function(req, res, next) { res.write('<h1>Middleware handler style</h1>'); next(); }); 

Application structure

The server code is not limited to handlers; an application can also contain a domain model, specialized libraries and utilities used in many processors, and other “places” for placing logic and data. All applications running in IAS are placed in the / applications directory and have the following structure:

• / app - root directory of handlers corresponding to the site root hostname hostname ,
• / config - application configuration,
• / doc - documentation and supporting materials
• / files - directory for placing user-uploaded files (2 or 3-level system of subdirectories are automatically built in it in order not to overload the file system with a large number of files),
• / init - initialization code that is started when the server starts (here you can programmatically create handlers, prepare data structures in memory, open tcp ports, etc.),
• / lib - directory for libraries and utilities that are loaded at startup (but after initialization) and accessible from all application code,
• / log - directory for logs (if the configuration is configured for separate logging for this application),
• / model - directory for domain models (also loaded at startup, but after initialization),
• / setup - js-files located in this directory will be launched only once when restarting IAS or the entire server, this place is necessary for update or migration scripts that are necessary to maintain the full life cycle of the application already during its operation,
• / tasks - directory for the placement of scheduled tasks, which are of two types: triggered, at a certain interval or at a designated time,
• / tmp - directory for temporary files.

In the next versions there will be more such directories ( issue # 195 ):

• / client - directory of the source of the client part,
• / static - collected clients will then be placed in /static , and several most common tools can be used as a collector.

IAS functionality

Let this article remain introductory, so I will not now describe in detail the entire arsenal of IAS and overload the reader. I will confine myself to a simple listing of the main one: registration with a service (daemon), transparent scaling for many processes and many servers, embedded system of users and sessions (including anonymous and authenticated), support for SSE (Server-Sent Events) and web sockets with the system channels and subscriptions to messages, support for query proxying, URL rewriting, network API introspection and issuing directory indexes, managing access to directories through access.js (similar to .htaccess), configuring applications, logging, log scrolling, article return and with caching into memory, gzip compression, HTTP support for "if-modified-since" and HTTP 304 (Not Modified), support for HTTPS, streaming files with support for return in parts (from the specified location to the specified location that players usually use For example, HTML5 video tag via HTTP Content-Range and Accept-Ranges headers, there are server quick deployment scripts for clean machines (CentOS, Ubuntu, Debian), built-in interprocess communication mechanisms via IPC, HTTP and ZeroMQ, a special API for state synchronization between processes, built-in server health monitoring mechanism, delayed task startup system, the ability to generate workers (parallel processes), validation of data structures and database schemas, generation of data structures from schemas for SQL-compatible DBMS, automatic error handling and long stack, optimization of garbage collection, sandbox screening (sandboxes), HTTP support basic authentication, processing virtual hosts and virtual paths, sticking IP (sticky), plugins (incl. passport, geoip, nodemailer, js minification, sass broadcast, etc.), unit testing subsystem, utilities for upload / download files and much more.

Conclusion

Impress (IAS) is actively developing, every week appears from 4 to 7 minor versions. Version 0.1.195 is currently relevant and version 0.2 is on the way, in which we will fix the application structure and basic API, observing backward compatibility for all 0.2.x versions. In 0.2.x we will deal only with optimization and error correction, and the expansion of functionality will be possible only if it does not require a redesign of applications based on 0.2.x. All major innovations and experiments will be introduced in parallel in the 0.3.x branch. I invite everyone to develop the project, and for my part I promise to support the code, at least, as long as it is relevant. Version 1.0 will only appear when I realize that independent developers are fully able to maintain the code. Now the documentation is being prepared, which until then was impossible due to the fact that the structure and architecture changed frequently, I will publish a link to it on readiness of version 0.2. Prior to this, you can learn more about IAS using examples that are installed with IAS as a default application.

Some numbers as of 2015-01-11: downloads from npm yesterday: 1,338, this week: 5,997, last month: 21,223, stars on github: 168, contribution to the repository: 8 people, lines of code: 6 120, source size: 207 Kb (of which kernel: 118Kb), average cyclomatic complexity of the code: 20, number of closed issues in github: 151, open issues: 9, date of first published version: 2013-06-08, in assemblies in Travis CI: 233, number of github commits: 468.

Links

NPM: www.npmjs.com/package/impress
Github: github.com/tshemsedinov/impress