Development of database service based on graphs: from prototype to reliable and scalable solution in the cloud
The databoom service is a cloud backend for Web and mobile applications. It reduces time and reduces the cost of prototyping and application development by simplifying data access. The following videos show how easily a prototype Web application is built using databoom.
Microsoft Azure makes it surprisingly easy to build robust, scalable applications. If a couple of years ago Azure meant Windows, then today Microsoft Azure is an excellent medium for Linux developers.
')
The databoom server part runs on Linux, uses Node.js and native modules written in C ++. The ease of deploying virtual machines, balancers, and other services under Azure has enabled us to quite easily deploy a cloud solution.
Starting from prototypes, many of our customers continue to work with us, creating ever more complex applications for which the issues of reliability, security and scalability are critical. Therefore, we chose Microsoft Azure.
The most important Azure benefits to us are the following:
To deploy the databoom solution on Azure, we chose a simple scheme (Figure 1).
Fig. 1. - Databoom cloud infrastructure architecture diagram
Here the numbers indicate the various levels of the system:
The load balancer distributes the load between the virtual machines and, in the event of an increase in the load, launches new virtual machines, providing automatic scaling.
For processing HTTP requests, single-core virtual machines with a Node.js application are used. Node.js by default uses one kernel. It would be possible to use multi-core computers with multiple instances of Node.js on each, but in this case we would have to connect the Cluster module on each machine. At the same time, the functions of the Cluster module are duplicated by the functions of the balancer, which means another extra layer in the process of passing the request. On the other hand, let's say, four single-core machines work no worse than one quad-core (in our tests, it turned out that by the number of requests per second, even better).
For interaction of multiple Node.js applications with a database, a specialized queue for large message flows was implemented. Each computer has several permanent TCP connections and asynchronously sends and receives message flows. Each TCP connection on each computer is capable of transmitting hundreds of thousands of requests per second. This allows the solution to be scaled to any desired number of requests per second.
Database management uses multi-core computers and a specialized load balancer. In the event of an increase in load, additional computers are raised. In case of failure of one of the computers, the database immediately rises on any least loaded or newly raised computer.
Load balancing, the ability to lift computers on the fly, multiple data duplication in storage, the ability to detect failures and transfer of load to other computers and many other features of MS Azure allowed us to create a reliable scalable solution.
Andrey Portnov, Victor Chernov, Vladislav Golovkov
Graduates of leading Moscow technical universities (MIPT and MEPI).
We have been dealing with data processing problems since the early 90s of the 20th century, when all three were employees of the computer center of the Atomic Energy Institute. I.Kurchatov (branch in Troitsk). We developed our own data processing technology, NitrosBase, which received several prizes at national and international exhibitions and forums. On the basis of this technology were developed DBMS class InMemory "NitrosBase InMemory DB" and graph database DBMS "NitrosBase RDF Storage". Based on NitrosBase RDF Storage, we built a databoom-BaaS service solution that greatly facilitates the development of Web and mobile applications.
Microsoft Azure makes it surprisingly easy to build robust, scalable applications. If a couple of years ago Azure meant Windows, then today Microsoft Azure is an excellent medium for Linux developers.
')
The databoom server part runs on Linux, uses Node.js and native modules written in C ++. The ease of deploying virtual machines, balancers, and other services under Azure has enabled us to quite easily deploy a cloud solution.
Starting from prototypes, many of our customers continue to work with us, creating ever more complex applications for which the issues of reliability, security and scalability are critical. Therefore, we chose Microsoft Azure.
The most important Azure benefits to us are the following:
- The ability to automatically increase the number of virtual machines with increasing load, and, consequently, their decrease when the load drops. In this case, the fee is taken only for really working machines.
- The ability to quickly automatically transfer the load to other machines in case of failure of one of the machines.
- Reliability of data storage. Each record on a virtual disk in Azure is copied to 3 different places in different racks at different ends of the data center.
To deploy the databoom solution on Azure, we chose a simple scheme (Figure 1).
Fig. 1. - Databoom cloud infrastructure architecture diagram
Here the numbers indicate the various levels of the system:
- Load balancer with scalability;
- Single-core virtual machines with a Node.js application that accepts HTTP requests, processes response receiving buffers, maintains simple logs, partially processes JSON data, parses OData requests, and so on;
- Specialized queue for large message flows.
- Database management machines.
- Reliable disk storage.
The load balancer distributes the load between the virtual machines and, in the event of an increase in the load, launches new virtual machines, providing automatic scaling.
For processing HTTP requests, single-core virtual machines with a Node.js application are used. Node.js by default uses one kernel. It would be possible to use multi-core computers with multiple instances of Node.js on each, but in this case we would have to connect the Cluster module on each machine. At the same time, the functions of the Cluster module are duplicated by the functions of the balancer, which means another extra layer in the process of passing the request. On the other hand, let's say, four single-core machines work no worse than one quad-core (in our tests, it turned out that by the number of requests per second, even better).
For interaction of multiple Node.js applications with a database, a specialized queue for large message flows was implemented. Each computer has several permanent TCP connections and asynchronously sends and receives message flows. Each TCP connection on each computer is capable of transmitting hundreds of thousands of requests per second. This allows the solution to be scaled to any desired number of requests per second.
Database management uses multi-core computers and a specialized load balancer. In the event of an increase in load, additional computers are raised. In case of failure of one of the computers, the database immediately rises on any least loaded or newly raised computer.
Load balancing, the ability to lift computers on the fly, multiple data duplication in storage, the ability to detect failures and transfer of load to other computers and many other features of MS Azure allowed us to create a reliable scalable solution.
About the authors
Andrey Portnov, Victor Chernov, Vladislav Golovkov
Graduates of leading Moscow technical universities (MIPT and MEPI).
We have been dealing with data processing problems since the early 90s of the 20th century, when all three were employees of the computer center of the Atomic Energy Institute. I.Kurchatov (branch in Troitsk). We developed our own data processing technology, NitrosBase, which received several prizes at national and international exhibitions and forums. On the basis of this technology were developed DBMS class InMemory "NitrosBase InMemory DB" and graph database DBMS "NitrosBase RDF Storage". Based on NitrosBase RDF Storage, we built a databoom-BaaS service solution that greatly facilitates the development of Web and mobile applications.
Source: https://habr.com/ru/post/273263/
All Articles