Data synchronization in real-time applications with Theron
Sometimes I draw myself a graph of how the architecture of modern systems should look and find those aspects of the development process that can be improved and those practices that can be applied to improve these processes. After another such iteration, I was once again convinced that there are amazing frameworks and methodologies for developing both server and client parts, but the synchronization of data between the client, server and database does not work in the way that modern realities require: fast response to changes in the system state, distribution and asynchronous processing of data, reuse of previously processed data.
In recent years, requirements for modern applications and methods for their development have changed significantly. Most of these applications use an asynchronous model consisting of many loosely coupled components (microservices). Users want the application to work smoothly and always be up to date (data should be synchronized at any time), in other words, users feel more comfortable when they don’t need to click the Refresh button every time or reload the application completely, if something went wrong. Under the cut there is a little theory and practice and a full-fledged open source application with React, Redux / Saga, Node, TypeScript and our project Theron.
Rick and Morty. Rick opens many portals.
I used various services to synchronize and store real-time data, most of which are mentioned in this article . But every time the application developed into a more complex product, it became obvious that the system depended too much on one service provider and did not have the necessary flexibility that the creation of its micro-architecture with a multitude of diversified satellite services gives it ( SQL and NoSQL) and writing code, instead of designers and control panels BaaS. Such an approach is indeed more complicated at the initial stage of prototype development, but it pays for itself in the future.
')
The result of my research was Theron. Theron is a service for creating modern real-time applications. Theron’s reactive data repository continuously transmits changes to the database based on a request to it. In just more than four months, we have implemented a basic architecture with a small team of two developers, the main criteria of which are:
I liked the functional approach even when I met one of the oldest functional programming languages focused on symbolic calculations Refal . Later, without realizing it, I began to use the reactive programming paradigm, and, over time, most of my work was built on these approaches.
Theron is based on ReactiveX . Theron's fundamental concept is reactive channels that provide a flexible way to broadcast data to different user segments. Theron uses the classic Pub / Sub design pattern . To create a new channel (unlimited amount) and streaming data, you just need to create a new subscription.
After installation , import Theron and create a new client:
Creating a new client does not establish a new WebSocket connection and does not begin to synchronize data. A connection is established only when a subscription is created, provided that there is no other active connection. That is, in the context of reactive programming, the Theron client and the channels are cold observable objects.
Create a new subscription:
When the channel is no longer needed - accomplish your goal:
Sending data to clients subscribed to this channel from the server side (Node.js) is also simple:
Theron uses exponential backoff (enabled by default) when connection is lost or when non-critical errors occur (English) : errors when a repeated subscription to the channel is possible.
The implementation of many algorithms in the framework of reactive programming is elegant and simple, for example, the exponential backoff in the Theron client library looks like this:
As mentioned above, Theron is a reactive data repository: a change notification system that continuously transmits updates via secure channels for your application, based on the usual SQL query to the database. Theron analyzes the database query and sends data artifacts that can be used to recreate the original data.
Consider how it works on the example of the life cycle of a simple list consisting of the first three elements, ordered alphabetically:
Before we continue, connect the database to Theron by entering data to access it in the control panel:
The internal structure of locking a database is a big topic for a separate article in the future. Theron is a distributed system, so the pool of connections to the database is limited to the 10th (with the possibility of increasing to 20) by common connections.
1. Creating a new subscription
Theron works with SQL queries, so your server should not return the result of the query, but the original SQL query. For example, in our case, the JSON response of the server might look like this:
On the client side, we will begin translating the data for our SQL query by creating a new subscription:
Theron will send a GET request '/ todos' to your server, check the validity of the returned SQL query and start translating the initial instructions with the necessary data, if this query has not been previously cached on the client side.
The TheronRowArtefact instruction is a regular JavaScript object with the `payload 'data itself and the` type` instruction type. The main types of instructions:
Suppose that several elements A , B , C already exist in the database. Then the change in the state of the client can be represented as follows (on the left - it was, on the right - it became):
Theron instructions for this state:
Each synchronization block starts and ends with the BEGIN_TRANSACTION and COMMIT_TRANSACTION instructions. For the correct sorting of items on the client side, Theron additionally sends data about the previous item.
2. User renames A (1) to D (1)
Suppose the user renames A (1) to D (1) . Since the SQL query arranges the elements in alphabetical order, the elements will be sorted, and the client state will change as follows:
Theron instructions for this state:
3. User creates new item A (4)
Suppose a user creates a new item A (4) . Since our SQL query limits the data to the first three elements, on the client side, the D (1) element will be deleted, and the client state will change as follows:
Theron instructions for this state:
4. User deletes element D (1)
Suppose the user deletes the D (1) entry from the database. In this case, Theron will not send new instructions, since this change in the database does not affect the data returned by our SQL query, and therefore does not affect the state of the client:
Client-side processing of instructions
Now, knowing how Theron works with data, we can implement logic to recreate the data on the client side. The algorithm is quite simple: we will use the type of instruction and the metadata of the previous element for the correct positioning of the elements in the array. In a real application, you need to use, for example, the Immutable.js library for working with arrays and the scan operator as an example .
Sometimes it is better to study, based on ready-made examples: therefore, here is the promised application published under the MIT license - https://github.com/therondb/figure . Figure is a service for working with HTML forms in static sites; The development strategy is React, Redux / Saga, Node, TypeScript and, of course, Theron. For example, we use Figure to create a list of subscribers to our blog and documentation site ( https://github.com/therondb/therondb.com ):
In addition to correcting a hypothetical ton of errors and the classic writing of client libraries for popular platforms, we are working on separating the reverse proxy server and the balancer into an independent component. The idea is to create an API on the server side, which can be accessed both through regular HTTP requests and through a permanent WebSocket connection. In the following article about Theron architecture I will write about it in more detail.
Our team is small but energetic, and we love to experiment. Theron is in active development: there are many ideas and points that need to be implemented and improved. We will be happy to hear any criticism, take advice and discuss it constructively.
In recent years, requirements for modern applications and methods for their development have changed significantly. Most of these applications use an asynchronous model consisting of many loosely coupled components (microservices). Users want the application to work smoothly and always be up to date (data should be synchronized at any time), in other words, users feel more comfortable when they don’t need to click the Refresh button every time or reload the application completely, if something went wrong. Under the cut there is a little theory and practice and a full-fledged open source application with React, Redux / Saga, Node, TypeScript and our project Theron.
Rick and Morty. Rick opens many portals.
I used various services to synchronize and store real-time data, most of which are mentioned in this article . But every time the application developed into a more complex product, it became obvious that the system depended too much on one service provider and did not have the necessary flexibility that the creation of its micro-architecture with a multitude of diversified satellite services gives it ( SQL and NoSQL) and writing code, instead of designers and control panels BaaS. Such an approach is indeed more complicated at the initial stage of prototype development, but it pays for itself in the future.
')
The result of my research was Theron. Theron is a service for creating modern real-time applications. Theron’s reactive data repository continuously transmits changes to the database based on a request to it. In just more than four months, we have implemented a basic architecture with a small team of two developers, the main criteria of which are:
- The rapid creation of new applications and painless migration of existing ones on Theron.
- The use of modern practices in the creation of asynchronous, distributed and fault-tolerant systems and the isomorphism of system components.
- Distributed low-level integration with databases such as Postgres and Mongo.
- Easy integration with modern frameworks such as React, Angular and their friends: ReactiveX, Redux etc.
- Focus on solving the problem of data synchronization, rather than providing a complete development stack and the subsequent " locking vendor ".
- The basic application logic (including authentication and access rights) should be implemented by developers on their side.
Reactive Channels
I liked the functional approach even when I met one of the oldest functional programming languages focused on symbolic calculations Refal . Later, without realizing it, I began to use the reactive programming paradigm, and, over time, most of my work was built on these approaches.
Theron is based on ReactiveX . Theron's fundamental concept is reactive channels that provide a flexible way to broadcast data to different user segments. Theron uses the classic Pub / Sub design pattern . To create a new channel (unlimited amount) and streaming data, you just need to create a new subscription.
After installation , import Theron and create a new client:
import { Theron } from 'theron'; const theron = new Theron('https://therondb.com', { app: 'YOUR_APP_NAME' }); Creating a new client does not establish a new WebSocket connection and does not begin to synchronize data. A connection is established only when a subscription is created, provided that there is no other active connection. That is, in the context of reactive programming, the Theron client and the channels are cold observable objects.
Create a new subscription:
const channel = theron.join('the-world'); const subscription = channel.subscribe( message => { console.log(message.payload); }, err => { console.log(err); }, () => { console.log('done'); } ); When the channel is no longer needed - accomplish your goal:
subscription.unsubscribe(); Sending data to clients subscribed to this channel from the server side (Node.js) is also simple:
import { Theron } from 'theron'; const theron = new Theron('https://therondb.com', { app: 'YOUR_APP_NAME', secret: 'YOUR_SECRET_KEY' }); theron.publish('the-world', { message: 'Greatings from Cybertron!' }).subscribe( res => { console.log(res); }, err => { console.log(err); }, () => { console.log('done'); }, ); Theron uses exponential backoff (enabled by default) when connection is lost or when non-critical errors occur (English) : errors when a repeated subscription to the channel is possible.
The implementation of many algorithms in the framework of reactive programming is elegant and simple, for example, the exponential backoff in the Theron client library looks like this:
let attemp = 0; const rescueChannel = channel.retryWhen(errs => errs.switchMap(() => Observable.timer(Math.pow(2, attemp + 1) * 500).do(() => attemp++)) ).do(() => attemp = 0); Database Integration
As mentioned above, Theron is a reactive data repository: a change notification system that continuously transmits updates via secure channels for your application, based on the usual SQL query to the database. Theron analyzes the database query and sends data artifacts that can be used to recreate the original data.
Theron is currently integrated with Postgres; integration with Mongo in the development process.
Consider how it works on the example of the life cycle of a simple list consisting of the first three elements, ordered alphabetically:
Before we continue, connect the database to Theron by entering data to access it in the control panel:
The internal structure of locking a database is a big topic for a separate article in the future. Theron is a distributed system, so the pool of connections to the database is limited to the 10th (with the possibility of increasing to 20) by common connections.
1. Creating a new subscription
Theron works with SQL queries, so your server should not return the result of the query, but the original SQL query. For example, in our case, the JSON response of the server might look like this:
{ "query": "SELECT * FROM todos ORDER BY name LIMIT 3" } On the client side, we will begin translating the data for our SQL query by creating a new subscription:
import { Theron } from 'theron'; const theron = new Theron('https://therondb.com', { app: 'YOUR_APP_NAME' }); const subscription = theron.watch('/todos').subscribe( action => { console.log(action); // Theron' }, err => { console.log(err); }, () => { console.log('complete'); } ); Theron will send a GET request '/ todos' to your server, check the validity of the returned SQL query and start translating the initial instructions with the necessary data, if this query has not been previously cached on the client side.
The TheronRowArtefact instruction is a regular JavaScript object with the `payload 'data itself and the` type` instruction type. The main types of instructions:
- ROW_ADDED - added new item.
- ROW_REMOVED - the item has been deleted.
- ROW_MOVED - the item has been changed.
- ROW_CHANGED - the item has been changed.
- BEGIN_TRANSACTION - new sync block.
- COMMIT_TRANSACTION - synchronization completed successfully.
- ROLLBACK_TRANSACTION - An error occurred during synchronization.
Suppose that several elements A , B , C already exist in the database. Then the change in the state of the client can be represented as follows (on the left - it was, on the right - it became):
| Id | Name | Id | Name |
|---|---|---|---|
| one | A | ||
| 2 | B | ||
| 3 | C |
Theron instructions for this state:
{ type: BEGIN_TRANSACTION }{ type: ROW_ADDED , payload: { row: { id: 1, name: 'A' }, prevRowId: null } }{ type: ROW_ADDED , payload: { row: { id: 2, name: 'B' }, prevRowId: 1 }{ type: ROW_ADDED , payload: { row: { id: 3, name: 'C' }, prevRowId: 2 } }{ type: BEGIN_TRANSACTION }
Each synchronization block starts and ends with the BEGIN_TRANSACTION and COMMIT_TRANSACTION instructions. For the correct sorting of items on the client side, Theron additionally sends data about the previous item.
2. User renames A (1) to D (1)
Suppose the user renames A (1) to D (1) . Since the SQL query arranges the elements in alphabetical order, the elements will be sorted, and the client state will change as follows:
| Id | Name | Id | Name |
|---|---|---|---|
| one | A | 2 | B |
| 2 | B | 3 | C |
| 3 | C | one | D |
Theron instructions for this state:
{ type: BEGIN_TRANSACTION }{ type: ROW_CHANGED , payload: { row: { id: 1, name: 'D' }, prevRowId: 3 } }{ type: ROW_MOVED , payload: { row: { id: 1, name: 'D' }, prevRowId: 3 } }{ type: ROW_MOVED , payload: { row: { id: 2, name: 'B' }, prevRowId: null } }{ type: ROW_MOVED , payload: { row: { id: 3, name: 'C' }, prevRowId: 2 } }{ type: COMMIT_TRANSACTION }
3. User creates new item A (4)
Suppose a user creates a new item A (4) . Since our SQL query limits the data to the first three elements, on the client side, the D (1) element will be deleted, and the client state will change as follows:
| Id | Name | Id | Name |
|---|---|---|---|
| 2 | B | four | A |
| 3 | C | 2 | B |
| one | D | 3 | C |
Theron instructions for this state:
{ type: BEGIN_TRANSACTION }{ type: ROW_ADDED , payload: { row: { id: 4, name: 'A' }, prevRowId: null } }{ type: ROW_MOVED , payload: { row: { id: 2, name: 'B' }, prevRowId: 4 } }{ type: ROW_MOVED , payload: { row: { id: 3, name: 'C' }, prevRowId: 2 } }{ type: ROW_REMOVED , payload: { row: { id: 1, name: 'D' }, prevRowId: 3 } }{ type: COMMIT_TRANSACTION }
4. User deletes element D (1)
Suppose the user deletes the D (1) entry from the database. In this case, Theron will not send new instructions, since this change in the database does not affect the data returned by our SQL query, and therefore does not affect the state of the client:
| Id | Name | Id | Name |
|---|---|---|---|
| four | A | four | A |
| 2 | B | 2 | B |
| 3 | C | 3 | C |
Client-side processing of instructions
Now, knowing how Theron works with data, we can implement logic to recreate the data on the client side. The algorithm is quite simple: we will use the type of instruction and the metadata of the previous element for the correct positioning of the elements in the array. In a real application, you need to use, for example, the Immutable.js library for working with arrays and the scan operator as an example .
import { ROW_ADDED, ROW_CHANGED, ROW_MOVED, ROW_REMOVED } from 'theron'; let todos = []; const subscription = theron.watch('/todos').subscribe( action => { switch (action.type) { case ROW_ADDED: const index = nextIndexForRow(rows, action.prevRowId) if (index !== -1) { rows.splice(index, 0, action.row); } break; case ROW_CHANGED: const index = indexForRow(rows, action.row.id); if (index !== -1) { rows[index] = action.row; } break; case ROW_MOVED: const index = indexForRow(rows, action.row.id); if (index !== -1) { const row = list.splice(curPos, 1)[0]; const newIndex = nextIndexForRow(rows, action.prevRowId); rows.splice(newIndex, 0, row); } break; case ROW_REMOVED: const index = indexForRow(rows, action.row.id); if (index !== -1) { list.splice(index, 1); } break; } }, err => { console.log(err); } ); function indexForRow(rows, rowId) { return rows.findIndex(row => row.id === rowId); } function nextIndexForRow(rows, prevRowId) { if (prevRowId === null) { return 0; } const index = indexForRow(rows, prevRowId); if (index === -1) { return rows.length; } else { return index + 1; } } Time examples
Sometimes it is better to study, based on ready-made examples: therefore, here is the promised application published under the MIT license - https://github.com/therondb/figure . Figure is a service for working with HTML forms in static sites; The development strategy is React, Redux / Saga, Node, TypeScript and, of course, Theron. For example, we use Figure to create a list of subscribers to our blog and documentation site ( https://github.com/therondb/therondb.com ):
Conclusion
In addition to correcting a hypothetical ton of errors and the classic writing of client libraries for popular platforms, we are working on separating the reverse proxy server and the balancer into an independent component. The idea is to create an API on the server side, which can be accessed both through regular HTTP requests and through a permanent WebSocket connection. In the following article about Theron architecture I will write about it in more detail.
Our team is small but energetic, and we love to experiment. Theron is in active development: there are many ideas and points that need to be implemented and improved. We will be happy to hear any criticism, take advice and discuss it constructively.
Source: https://habr.com/ru/post/303436/
All Articles