Logux: Connection lost, data synchronized - interview with Andrey Sitnik (Evil Martians)
What happens if the Internet suddenly disappears on the client? Perhaps for a split second, or maybe for a more tangible period? We all as users are faced with an unstable signal, a floating quality of communication. Sometimes it doesn't matter, because you want to watch some funny video, sometimes a lot can depend on it - imagine that you urgently need to buy a train ticket or pay for the same Internet.
Those areas where the end user is very important - for example, the media, say that already 13% of users leave if your site opens for more than four seconds without understanding the reasons. And now let's imagine such a user who is still trying to send a comment, and he constantly “falls off” due to communication problems?
The percentage of withdrawals and failures will certainly be more. How to avoid it? What can be done in a situation where the data must be guaranteed to be sent both from the client and from the server?
')
This and other questions are answered by Andrey Sitnik - the author of PostCSS and Autoprefixer, leading the fender in “Evil Martians”.
“Why do we even talk about communication problems?” Is this not a question solely related to physical / network capabilities?
- OSI is not quite the topic here. Logux replaces REST and AJAX. That is, it is a purely applied level. Logux solves the following problems:
To do this, Logux creates a special virtual event channel between the client and the server.
The client can put events on this channel - Logux will send them when it is available. And the same on the server. Let's sort it out at the technical level step by step:
1. On the client, this will be a one-to-one library with APIs like Redux (the React is not tied, there is a lower-level API). You create action in the same way. But for some actions you can set the key sync: true - then Logux will deliver them to the server itself. This library holds a web socket, sends a ping, checking that there is a connection.
- So, why is it good that this library is not tied to React?
Not everyone develops on React (and this is very good for a variety of environments). Someone might use Vue.js, Angular. Or just have a JS application where there is no HTML, so the React will not be needed so much. Go ahead:
2. Additional library, which, when communication is lost, will show specials. “No Link” badge at the top of the page and will change the favicon. If there are events in the channel for sending, but there is no connection, she will also tell the user that not all the data has been saved.
- Why does the user need to know this? For which interaction format can this be critically important?
- If the user gave us some data, we must notify him that they have not been saved. I would say that this behavior should be the default. The library is optional in many ways in order to write such a notification yourself and better integrate it into the design. Although there are cases where you can skip connection problems - for example, if Logux is used to collect user actions for analytics.
Next step.
3. When the connection appears, the Logux client and the server will send each other the events that were added during the absence of the connection.
- It is clear, it is logical, how many actions can be in the queue?
- Usually 10-20 events. But we have no hard limits - how much will fit in the memory of the client and server.
- The order of performance of these events?
- The order of events is controlled very strictly. The problem is not only in the order of events of one user. But even when several users are working, the order of all their events must be the same on all systems (so that the final state is the same). Therefore, in Logux, each event is assigned a creation time. It is quite tricky - for example, the difference in time between the client and the server is taken into account.
- Could it be that some events no longer have to be performed? Mutual verification of action dependencies?
- The client can clean the log from already unnecessary events. This is already determined by business logic.
Next step: the user can close the browser - unsent events will be saved in localStorage. But before attempting to close, the library will ask the user to first go online, as there is unsent data.
- Again we return to the issue of capacity and importance, and what happens if the user forcibly closes the browser?
- Nothing wrong. As soon as the user returns to the site, the data will be sent.
Now all the remaining steps:
5. To write a Logux server, there is a special server framework. That is, it is like express.js. You yourself describe how the server responds to a particular event from a client. At the same time, the logic is a bit more cunning - the fact that the event has come now does not mean that it was created now. Therefore, each event has a creation time. When connecting, the client and server determine the time difference and synchronize the clock (and the time of creating old events).
6. Server framework for now on node.js. Then we will do for Elixir and Go. But you can use Ruby, Python, and PHP — just put the node.js server as a proxy to hold the socket. And already this proxy server will send the old REST request to the ruby ​​server.
7. Since we have this event log and exact time, you can make CRDT on top of it like in Swarm.js - and automatically resolve conflicts in many simple cases.
8. In general, in most cases you can not draw a twister - the client can immediately draw as if the event has already run (for example, a comment has been sent), Logux will show the user that the data has not been saved to the server. Of course, there will be cases where this does not work out - for example, payment. There you can do the old logic with a twist.
- Thanks, and how is Logux better then the existing solutions?
- There is Relay / GraphQL - they reduce the amount of code when requesting data. But changing the data itself is also not very easy to implement. Live update is bad. Offline is also not worked out.
There are isomorphic databases - they work on the client and on the server and synchronize the data between themselves. For example, CouchDB and Firebase. It is well resolved live update data. Ready CRDT is now a few who have, but, in general, it can be implemented. But they are quite difficult to expand, many developers are afraid of such a strange approach to synchronization, so these databases did not become an industry standard.
There are also CRDT libraries - for example, Swarm.js. Logux copies many ideas from Swarm.js. But Swarm.js is difficult to make friends with React and Redax. It is also difficult to transfer non-CRDT operations in it.
- What are the strengths and weaknesses?
- Less code, live data update out of the box, easy to make an application to work offline. And all this with the familiar semantics of Redax.
The main weak point - you need to run an additional server. But all such decisions have it. Plus, you can make this server a proxy server and keep all the logic stored in PHP or Ruby on Rails.
- Where did the need for this dedicated solution come from?
- On the Amplifer it was necessary to show live statistics of publications and update the page with errors immediately when a problem appeared. And we realized that there was no good solution. We tried to implement Swarm.js, but it wasn’t easy to deploy. So I started thinking how to make friends with Swarm.js and Redax. In St. Petersburg, at that time, Dan Abramov was just, and the idea of ​​a separate library was born in a conversation with him.
- Does it have a chance to become an industry standard?
- I try to make Logux as a universal solution for all web applications. But, it seems to me, in 2017 we will see many more attempts to redo AJAX / REST - now this is the main problem of web development, in my opinion. Who will win this fight, we learn only in 2018.
- Are there 1-2 interesting practical case?
- Yes, even TODO MVC - you also want to continue working with the task list even if there is no connection. Or if you added a task to a computer, then you want it to appear right there on your phone, without reloading the pages.
Or comments - a live update, like on Facebook or VK, useful for engaging people.
There is one more advantage of Logux for any site - it is an “optimistic interface” ( another HolyJS speaker was just talking about it). Now, with every action we show the user a "twister". Each save locks the interface and disrupts the user's flow. With Logux, it is an order of magnitude easier to make an optimistic interface, where saving will occur in the background, distracting the user only if an error has indeed occurred. For example, this is how Google Inbox works. Any web application will benefit from an optimistic interface, as users always love fast interfaces.
- Thank you very much for the interview, and see you at the conference.
Of course, in an interview I managed to discuss only a general view of the library and its philosophy, Andrei will give a presentation on HolyJS with a detailed hourly report on the development and work with Logux. In addition, you can immediately see the reports on other discussed technologies:
Those areas where the end user is very important - for example, the media, say that already 13% of users leave if your site opens for more than four seconds without understanding the reasons. And now let's imagine such a user who is still trying to send a comment, and he constantly “falls off” due to communication problems?
The percentage of withdrawals and failures will certainly be more. How to avoid it? What can be done in a situation where the data must be guaranteed to be sent both from the client and from the server?
')
This and other questions are answered by Andrey Sitnik - the author of PostCSS and Autoprefixer, leading the fender in “Evil Martians”.“Why do we even talk about communication problems?” Is this not a question solely related to physical / network capabilities?
- OSI is not quite the topic here. Logux replaces REST and AJAX. That is, it is a purely applied level. Logux solves the following problems:
- Now it takes a lot of code for simple queries.
- Live update data from the server to write an order of magnitude more difficult.
- Good offline support is generally a hell of a write. But offline is always needed, since we always have “500 ms offline” - these are constant buttons on buttons.
To do this, Logux creates a special virtual event channel between the client and the server.
The client can put events on this channel - Logux will send them when it is available. And the same on the server. Let's sort it out at the technical level step by step:
Debriefing
1. On the client, this will be a one-to-one library with APIs like Redux (the React is not tied, there is a lower-level API). You create action in the same way. But for some actions you can set the key sync: true - then Logux will deliver them to the server itself. This library holds a web socket, sends a ping, checking that there is a connection.
- So, why is it good that this library is not tied to React?
Not everyone develops on React (and this is very good for a variety of environments). Someone might use Vue.js, Angular. Or just have a JS application where there is no HTML, so the React will not be needed so much. Go ahead:
2. Additional library, which, when communication is lost, will show specials. “No Link” badge at the top of the page and will change the favicon. If there are events in the channel for sending, but there is no connection, she will also tell the user that not all the data has been saved.
- Why does the user need to know this? For which interaction format can this be critically important?
- If the user gave us some data, we must notify him that they have not been saved. I would say that this behavior should be the default. The library is optional in many ways in order to write such a notification yourself and better integrate it into the design. Although there are cases where you can skip connection problems - for example, if Logux is used to collect user actions for analytics.
Next step.
3. When the connection appears, the Logux client and the server will send each other the events that were added during the absence of the connection.
- It is clear, it is logical, how many actions can be in the queue?
- Usually 10-20 events. But we have no hard limits - how much will fit in the memory of the client and server.
- The order of performance of these events?
- The order of events is controlled very strictly. The problem is not only in the order of events of one user. But even when several users are working, the order of all their events must be the same on all systems (so that the final state is the same). Therefore, in Logux, each event is assigned a creation time. It is quite tricky - for example, the difference in time between the client and the server is taken into account.
- Could it be that some events no longer have to be performed? Mutual verification of action dependencies?
- The client can clean the log from already unnecessary events. This is already determined by business logic.
Next step: the user can close the browser - unsent events will be saved in localStorage. But before attempting to close, the library will ask the user to first go online, as there is unsent data.
- Again we return to the issue of capacity and importance, and what happens if the user forcibly closes the browser?
- Nothing wrong. As soon as the user returns to the site, the data will be sent.
Now all the remaining steps:
5. To write a Logux server, there is a special server framework. That is, it is like express.js. You yourself describe how the server responds to a particular event from a client. At the same time, the logic is a bit more cunning - the fact that the event has come now does not mean that it was created now. Therefore, each event has a creation time. When connecting, the client and server determine the time difference and synchronize the clock (and the time of creating old events).
6. Server framework for now on node.js. Then we will do for Elixir and Go. But you can use Ruby, Python, and PHP — just put the node.js server as a proxy to hold the socket. And already this proxy server will send the old REST request to the ruby ​​server.
7. Since we have this event log and exact time, you can make CRDT on top of it like in Swarm.js - and automatically resolve conflicts in many simple cases.
8. In general, in most cases you can not draw a twister - the client can immediately draw as if the event has already run (for example, a comment has been sent), Logux will show the user that the data has not been saved to the server. Of course, there will be cases where this does not work out - for example, payment. There you can do the old logic with a twist.
Logux and alternatives
- Thanks, and how is Logux better then the existing solutions?
- There is Relay / GraphQL - they reduce the amount of code when requesting data. But changing the data itself is also not very easy to implement. Live update is bad. Offline is also not worked out.
There are isomorphic databases - they work on the client and on the server and synchronize the data between themselves. For example, CouchDB and Firebase. It is well resolved live update data. Ready CRDT is now a few who have, but, in general, it can be implemented. But they are quite difficult to expand, many developers are afraid of such a strange approach to synchronization, so these databases did not become an industry standard.
There are also CRDT libraries - for example, Swarm.js. Logux copies many ideas from Swarm.js. But Swarm.js is difficult to make friends with React and Redax. It is also difficult to transfer non-CRDT operations in it.
- What are the strengths and weaknesses?
- Less code, live data update out of the box, easy to make an application to work offline. And all this with the familiar semantics of Redax.
The main weak point - you need to run an additional server. But all such decisions have it. Plus, you can make this server a proxy server and keep all the logic stored in PHP or Ruby on Rails.
- Where did the need for this dedicated solution come from?
- On the Amplifer it was necessary to show live statistics of publications and update the page with errors immediately when a problem appeared. And we realized that there was no good solution. We tried to implement Swarm.js, but it wasn’t easy to deploy. So I started thinking how to make friends with Swarm.js and Redax. In St. Petersburg, at that time, Dan Abramov was just, and the idea of ​​a separate library was born in a conversation with him.
- Does it have a chance to become an industry standard?
- I try to make Logux as a universal solution for all web applications. But, it seems to me, in 2017 we will see many more attempts to redo AJAX / REST - now this is the main problem of web development, in my opinion. Who will win this fight, we learn only in 2018.
Life examples
- Are there 1-2 interesting practical case?
- Yes, even TODO MVC - you also want to continue working with the task list even if there is no connection. Or if you added a task to a computer, then you want it to appear right there on your phone, without reloading the pages.
Or comments - a live update, like on Facebook or VK, useful for engaging people.
There is one more advantage of Logux for any site - it is an “optimistic interface” ( another HolyJS speaker was just talking about it). Now, with every action we show the user a "twister". Each save locks the interface and disrupts the user's flow. With Logux, it is an order of magnitude easier to make an optimistic interface, where saving will occur in the background, distracting the user only if an error has indeed occurred. For example, this is how Google Inbox works. Any web application will benefit from an optimistic interface, as users always love fast interfaces.
- Thank you very much for the interview, and see you at the conference.
Of course, in an interview I managed to discuss only a general view of the library and its philosophy, Andrei will give a presentation on HolyJS with a detailed hourly report on the development and work with Logux. In addition, you can immediately see the reports on other discussed technologies:
Source: https://habr.com/ru/post/316732/
All Articles