We invent technology: the experience of connecting several open-source projects
I would like to share the experience of creating an application, or more precisely a technology based on several open-source projects. This is something like a web server with support for asynchronous messages and the ability to execute scripts in JavaScript.
I encountered the need to invent something after the end of several projects related somehow to the automation of finance. We used the usual three-tier model client-server application-database server with some modifications. I realized that even a bunch of improvements does not make this technology convenient. She always “crawls out” and needs attention.
Thus, I thought about the “ideal” automation model of something and came to some conclusions.
')
1) The client must be a browser.
Only a very serious reason can force us to make a “fat” application with its graphic library and then as an add-on to the browser. Having looked at the current state of affairs, at the speed of Chrome, at Google Gears, Google Docs, at work with the canvas object, at the possibility of graphical frameworks in JavaScript, I was firmly established in this decision.
2) The most convenient form of data storage is a relational database with SQL.
Everywhere! That is, not only also on the application server, but also on the client! There is no rational reason to drop the ability to have data in a relational form, even for a client, except for the need for wild performance. Of course, by this time I had already stumbled upon SQLITE ( sqlite.org ). This is a built-in cross-platform relational database in source codes, in the public domain, with support for virtually everything that is really needed in SQL. I generally advise anyone to get acquainted with this project as a training tool. SQLITE has very interesting (and useful in my opinion) features. For example. The data type is stored together with the value itself, that is, in the same column of the table in different records, you can store both a string, a number, and a blob. It is possible to create an in-memory database (that is, completely in RAM). There are triggers. There is no built-in language! The source code for the project is available in a convenient form called amalgamation ( sqlite.org/amalgamation.html ), that is, literally from two files: cpp and h. It is very convenient to include in the project. The project is flexibly configured compilation options.
3) Business logic should be created in a high-level scripting language.
It should not be C ++ or Java. These languages ​​will introduce unnecessary complexity and will not give the necessary flexibility. Oddly enough, my choice fell on JavaScript. At one time I was doing quite a bit of web programming and I had a very pleasant impression of the JavaScript language. This is a very flexible and easily extensible language, but for some reason it is not used as a server solution. As the implementation, I chose Google V8 ( code.google.com/p/v8 ). This is a compiler and JavaScript runtime project that implements the ECMA-262 standard ( www.ecma-international.org/publications/standards/Ecma-262.htm ) and is used in the Chrome browser.
V8 source code is available through the subversion system ( ru.wikipedia.org/wiki/Subversion ). Often subversion is called svn by the name of a standard client application. In the wiki, everything is well described, but in a nutshell, this is a system of repositories (centralized file repositories along with their entire history of changes) and working copies (“snapshots” of project branches in the form of directories and files on users' local computers). This system has replaced CVS. After the trial, I chose SmartSVN ( www.syntevo.com/smartsvn ) as a subversion client and was more or less satisfied. There is a free version.
4) The transport of the future system should be based on asynchronous messages.
This will allow applications to forget about the intricacies of connections and protocol nuances, to create a universal layer for sending messages. The transport layer acquires a familiar event form.
We must get to work, but where to start? I don’t really want to work with files, network, even with cross-platform capabilities from scratch, why refuse Linux as an alternative, and maybe the main platform? After a brief search, I found STLPlus ( stlplus.sourceforge.net ). This is a fairly well-designed Windows and Unix C ++ library, which has everything you need for the initial set of functionality. I had to drank it with a file.
So, first of all create an asynchronous messaging system. We will not be wasted on trifles, because we are building an ideal. Therefore, this system should be a) zero-configuration and b) automatically switch from a faulty network to a healthy one. This means that every computer within our network must send out UDP broadcast messages with a list of its IP addresses all the time, and as a result, each computer will communicate with each one through all available networks and exclude the connection used from the exchange in the event of a network failure. In the simplest case, we can connect computers using Ethernet and WiFi, and if the Ethernet fails, the system automatically switches to WiFi. Access control issues will be postponed for now.
The next layer implements addressing. Here is the simplest solution. On each computer, you can register an arbitrary network address (point @ host) and send / receive messages. Plus, there is a system like message boards in fido, which allows one network address to declare the processing of certain names, and another to look for these handlers. This allows you to freely move tasks across computers without any need for configuration changes.
The next step is to embed a web server. Unfortunately, I did not find anything suitable in open-source. Either the solutions were too monstrous, or narrowly specialized, or too simplified, or they dragged their libraries with them. I had to write this part, that is, support HTTP 1.1 as a server, on my own. There is nothing special here, everything is well documented, checked by the browser right there.
All tasks our application performs within sessions. This is something that is created on the server to handle client requests or to run constant background tasks. As a session, our system uses ... a javascript context. Figuratively, this can be represented as follows: when a client opens a document in the browser and makes a request, a certain “document” also opens on the server, inside which javaScript code is executed to process requests. An astute reader probably already guessed that in this “document” there is no complete implementation of the DOM or, say, a window object. Of course, but there are much more useful wrappers: access to the messaging system and the SQLITE database. Moreover, in-memory SQLITE database is created automatically for each new session. The ability to send the result of any SELECT query within the message is built into the messaging system and this data will be automatically restored in the recipient database. That is, the script actually works all the time using SQL from a local relational database, into which you can retrieve any data from a remote database.
However, the javascript context, of course, allows you to do more than that. After all, in fact, we are dealing with a global javaScript object, we have the opportunity to freely create new properties for it, define functions, that is, do everything the same as in the client context in a browser window. We cache any required data in properties, etc. And virtually all the javaScript code being executed will be compiled before execution, the performance is very high.
Let's return to our client: the browser. Recently, there has been a trend of turning web pages into web applications (read the first sentence in the Chrome comic, www.google.com/googlebooks/chrome/small_00.html ). And what does it mean? This means that the processing of data presentation is being transferred more and more to the client, that is, to the browser. In fact, at present, nothing prevents to form visible HTML entirely on the browser. A modern web page will first load the necessary styles, bile libraries (in fact, the only time), and then go offline and allow the user to customize the look and decompose the visual elements as needed. Access to the web server will occur only for the actual data, say, in the form of XML. Of course, I draw a certain ideal, but everything is now striving for it. And most importantly, technology makes it possible.
Let's say Google Gears ( gears.google.com , the wiki is here: ru.wikipedia.org/wiki/Google_Gears ). This is the ability of the browser to work with the local SQLITE database, cache resources, and run asynchronous background javascript execution threads.
To move data from the SQLITE database to the browser's SQLITE database, I wrote the simplest replication based on triggers. All you need to do is to list the replicated tables and periodically retrieve SQL changes of the remote database (s). Data is collected using AJAX requests.
Here I will make a small digression about AJAX. Actually, I think he is known to all. But, faced with the need to embed it in client libraries, I was surprised to find that almost all AJAX examples on the Internet are horrible in terms of design. The same bewilderment befell me when I wanted to quickly find the simplest script engine to send the form via an AJAX POST request. Then, of course, I already guessed that most programmers today use certain frameworks (such as jQuery) on the client, which have similar functionality built into them.
Actually, summing up the line. Three months later, the system lives and feels great. All ideas are implemented in full. In general, I was pleasantly surprised that it all worked and almost as intended. There were also disappointments, the most significant of which probably should be considered the limitation of the V8, which in fact is not multi-threaded. The code can be executed at a time by only one thread, although you can prepare as many contexts as you like. Chrome runs a separate process on each tab.
System performance is enough for us. Actually, now we start its operation. On the results, if interested, I will inform later in a separate article.
I encountered the need to invent something after the end of several projects related somehow to the automation of finance. We used the usual three-tier model client-server application-database server with some modifications. I realized that even a bunch of improvements does not make this technology convenient. She always “crawls out” and needs attention.
Thus, I thought about the “ideal” automation model of something and came to some conclusions.
')
1) The client must be a browser.
Only a very serious reason can force us to make a “fat” application with its graphic library and then as an add-on to the browser. Having looked at the current state of affairs, at the speed of Chrome, at Google Gears, Google Docs, at work with the canvas object, at the possibility of graphical frameworks in JavaScript, I was firmly established in this decision.
2) The most convenient form of data storage is a relational database with SQL.
Everywhere! That is, not only also on the application server, but also on the client! There is no rational reason to drop the ability to have data in a relational form, even for a client, except for the need for wild performance. Of course, by this time I had already stumbled upon SQLITE ( sqlite.org ). This is a built-in cross-platform relational database in source codes, in the public domain, with support for virtually everything that is really needed in SQL. I generally advise anyone to get acquainted with this project as a training tool. SQLITE has very interesting (and useful in my opinion) features. For example. The data type is stored together with the value itself, that is, in the same column of the table in different records, you can store both a string, a number, and a blob. It is possible to create an in-memory database (that is, completely in RAM). There are triggers. There is no built-in language! The source code for the project is available in a convenient form called amalgamation ( sqlite.org/amalgamation.html ), that is, literally from two files: cpp and h. It is very convenient to include in the project. The project is flexibly configured compilation options.
3) Business logic should be created in a high-level scripting language.
It should not be C ++ or Java. These languages ​​will introduce unnecessary complexity and will not give the necessary flexibility. Oddly enough, my choice fell on JavaScript. At one time I was doing quite a bit of web programming and I had a very pleasant impression of the JavaScript language. This is a very flexible and easily extensible language, but for some reason it is not used as a server solution. As the implementation, I chose Google V8 ( code.google.com/p/v8 ). This is a compiler and JavaScript runtime project that implements the ECMA-262 standard ( www.ecma-international.org/publications/standards/Ecma-262.htm ) and is used in the Chrome browser.
V8 source code is available through the subversion system ( ru.wikipedia.org/wiki/Subversion ). Often subversion is called svn by the name of a standard client application. In the wiki, everything is well described, but in a nutshell, this is a system of repositories (centralized file repositories along with their entire history of changes) and working copies (“snapshots” of project branches in the form of directories and files on users' local computers). This system has replaced CVS. After the trial, I chose SmartSVN ( www.syntevo.com/smartsvn ) as a subversion client and was more or less satisfied. There is a free version.
4) The transport of the future system should be based on asynchronous messages.
This will allow applications to forget about the intricacies of connections and protocol nuances, to create a universal layer for sending messages. The transport layer acquires a familiar event form.
We must get to work, but where to start? I don’t really want to work with files, network, even with cross-platform capabilities from scratch, why refuse Linux as an alternative, and maybe the main platform? After a brief search, I found STLPlus ( stlplus.sourceforge.net ). This is a fairly well-designed Windows and Unix C ++ library, which has everything you need for the initial set of functionality. I had to drank it with a file.
So, first of all create an asynchronous messaging system. We will not be wasted on trifles, because we are building an ideal. Therefore, this system should be a) zero-configuration and b) automatically switch from a faulty network to a healthy one. This means that every computer within our network must send out UDP broadcast messages with a list of its IP addresses all the time, and as a result, each computer will communicate with each one through all available networks and exclude the connection used from the exchange in the event of a network failure. In the simplest case, we can connect computers using Ethernet and WiFi, and if the Ethernet fails, the system automatically switches to WiFi. Access control issues will be postponed for now.
The next layer implements addressing. Here is the simplest solution. On each computer, you can register an arbitrary network address (point @ host) and send / receive messages. Plus, there is a system like message boards in fido, which allows one network address to declare the processing of certain names, and another to look for these handlers. This allows you to freely move tasks across computers without any need for configuration changes.
The next step is to embed a web server. Unfortunately, I did not find anything suitable in open-source. Either the solutions were too monstrous, or narrowly specialized, or too simplified, or they dragged their libraries with them. I had to write this part, that is, support HTTP 1.1 as a server, on my own. There is nothing special here, everything is well documented, checked by the browser right there.
All tasks our application performs within sessions. This is something that is created on the server to handle client requests or to run constant background tasks. As a session, our system uses ... a javascript context. Figuratively, this can be represented as follows: when a client opens a document in the browser and makes a request, a certain “document” also opens on the server, inside which javaScript code is executed to process requests. An astute reader probably already guessed that in this “document” there is no complete implementation of the DOM or, say, a window object. Of course, but there are much more useful wrappers: access to the messaging system and the SQLITE database. Moreover, in-memory SQLITE database is created automatically for each new session. The ability to send the result of any SELECT query within the message is built into the messaging system and this data will be automatically restored in the recipient database. That is, the script actually works all the time using SQL from a local relational database, into which you can retrieve any data from a remote database.
However, the javascript context, of course, allows you to do more than that. After all, in fact, we are dealing with a global javaScript object, we have the opportunity to freely create new properties for it, define functions, that is, do everything the same as in the client context in a browser window. We cache any required data in properties, etc. And virtually all the javaScript code being executed will be compiled before execution, the performance is very high.
Let's return to our client: the browser. Recently, there has been a trend of turning web pages into web applications (read the first sentence in the Chrome comic, www.google.com/googlebooks/chrome/small_00.html ). And what does it mean? This means that the processing of data presentation is being transferred more and more to the client, that is, to the browser. In fact, at present, nothing prevents to form visible HTML entirely on the browser. A modern web page will first load the necessary styles, bile libraries (in fact, the only time), and then go offline and allow the user to customize the look and decompose the visual elements as needed. Access to the web server will occur only for the actual data, say, in the form of XML. Of course, I draw a certain ideal, but everything is now striving for it. And most importantly, technology makes it possible.
Let's say Google Gears ( gears.google.com , the wiki is here: ru.wikipedia.org/wiki/Google_Gears ). This is the ability of the browser to work with the local SQLITE database, cache resources, and run asynchronous background javascript execution threads.
To move data from the SQLITE database to the browser's SQLITE database, I wrote the simplest replication based on triggers. All you need to do is to list the replicated tables and periodically retrieve SQL changes of the remote database (s). Data is collected using AJAX requests.
Here I will make a small digression about AJAX. Actually, I think he is known to all. But, faced with the need to embed it in client libraries, I was surprised to find that almost all AJAX examples on the Internet are horrible in terms of design. The same bewilderment befell me when I wanted to quickly find the simplest script engine to send the form via an AJAX POST request. Then, of course, I already guessed that most programmers today use certain frameworks (such as jQuery) on the client, which have similar functionality built into them.
Actually, summing up the line. Three months later, the system lives and feels great. All ideas are implemented in full. In general, I was pleasantly surprised that it all worked and almost as intended. There were also disappointments, the most significant of which probably should be considered the limitation of the V8, which in fact is not multi-threaded. The code can be executed at a time by only one thread, although you can prepare as many contexts as you like. Chrome runs a separate process on each tab.
System performance is enough for us. Actually, now we start its operation. On the results, if interested, I will inform later in a separate article.
Source: https://habr.com/ru/post/48842/
All Articles