Development of a real-time synchronization system using SockJS, Django, Tornado and ZeroMQ

Not so long ago, while developing a new software product, our development team was faced with the task of implementing a full-fledged system for synchronizing user data in real time, by sending (PUSH) server changes. In the application itself, the amount of data was not great, but they could be viewed by several users simultaneously. Therefore, we needed a lightweight and fairly productive approach to data synchronization within a Web application. After we had considered various ways to solve this problem, we chose a fairly popular WebSocket emulator, SockJS, which uses different algorithms for data exchange between the client and the server, depending on the browser the client uses. Within this article, I will not focus on why this choice was made (quite a lot of articles were written on this subject, including on habrakhabr), but just to say that we have never regretted it.



Initially, when studying standard approaches to the implementation of such tasks, we faced one problem. This problem was that the interaction with our system was made not only through the web interface, but also through the use of API by third-party products that we could not control. And the end user of our product, of course, expects to see all the information about changes in the data that relate to it. The standard approach of using sockjs server implies that notifications about changes in any data in the system will be sent using the same JS client that is used to obtain information about these changes. That is why in our case such an approach would not be applicable.



In this article I would like to talk about how we solved this problem.



So, the source data:

')

• portal (written in Django) with API
• web client (with sockjs-client library for organizing synchronization)
• SockJS server (in the form of tornado-sockjs, which was one of the reasons why we made our choice in the direction of this product)

The overall task can be divided into two subtasks:

• The organization of receiving notifications by the end user about changes in the system (a trivial task)
• Creating messages / notifications about changes in the system using the Django tools (recall that we cannot use the standard approach and create such messages using the standard JS client, because in this case we will not be able to process the changes made using a set of APIs ).

Having studied the available information, it became clear that this task could not be solved without an “intermediary”. As an intermediary, a messaging system was chosen, namely ZeroMQ. Among the main advantages of this particular system, it is worth noting its simplicity, lightness, as well as support for the Python language (which is a very important condition for us).



In general, the resulting solution can be represented in the form of the scheme presented below.







Changes in the system data made by the end user using the web interface or APIs ultimately fall on our Django server, where ZeroMQ transfers the server to SockJS, which in turn notifies users of the fact that the data of the displayed object has changed.



Let's go to the code

Web client

To receive notifications, we need to connect the library sockjs-client, establish a connection to the server and subscribe to the necessary events. This can be done approximately as follows:

<script src="/js/sockjs-0.3.4.min.js"></script> <script> var SyncServer = new SockJS("http://example.com/echo"); SyncServer.onmessage = function(e) { //    }; </script> 

ZeroMQ server

To implement this part, we need ZeroMQ itself and the Python library to work with it - pyzmq. After that, you need to configure and start the server in proxy mode (this mode is called FORWARDER). This can be done in just a few lines of code:

 import zmq def main(): try: context = zmq.Context(1) # Socket facing clients frontend = context.socket(zmq.SUB) frontend.bind("tcp://127.0.0.1:XXXX") frontend.setsockopt(zmq.SUBSCRIBE, "") # Socket facing services backend = context.socket(zmq.PUB) backend.bind("tcp://127.0.0.1:YYYY") zmq.device(zmq.FORWARDER, frontend, backend) except Exception, e: # Handle exception pass finally: frontend.close() backend.close() context.term() if __name__ == '__main__': main() 

By running this script, you will receive a proxy server, waiting for messages on port XXXX and sending them to port YYYY.

SockJS server

A standard SockJS-tornado was chosen as the SockJS server with some minor changes that allow it to receive messages from the ZeroMQ server (you can find these changes in the __init__ method of the SockJSMyRouter class)

 from sockjs.tornado import SockJSConnection, SockJSRouter from tornado import ioloop as tornado_ioloop, web import zmq from zmq.eventloop import ioloop from zmq.eventloop.zmqstream import ZMQStream ioloop.install() io_loop = tornado_ioloop.IOLoop.instance() class SyncConnection(SockJSConnection): _connected = set() stream = None def __init__(self, session): super(SyncConnection, self).__init__(session) self.stream.on_recv(self.on_server_message) def on_open(self, request): self._connected.add(self) def on_message(self, data): pass def on_server_message(self, data): message = "your message" self.broadcast(self._connected, {'message': message}) def on_close(self): self._connected.remove(self) class SockJSMyRouter(SockJSRouter): def __init__(self, *args, **kw): super(SockJSMyRouter, self).__init__(*args, **kw) socket = context.socket(zmq.SUB) socket.setsockopt(zmq.SUBSCRIBE, "") socket.connect("tcp://127.0.0.1:YYYY") self._connection.stream = ZMQStream(socket, self.io_loop) if __name__ == '__main__': context = zmq.Context() EchoRouter = SockJSMyRouter(SyncConnection, '/echo') app = web.Application(EchoRouter.urls) app.listen("ZZZZ") io_loop.start() 

Django server

As part of the Django server, we need to add only a few lines responsible for creating notifications and sending them to our ZeroMQ server. You can do this as follows:

 import zmq context = zmq.Context() socket = context.socket(zmq.PUB) socket.connect("XXX") socket.send("your message") socket.close() 

That seems to be all! Having added the methods responsible for creating and processing messages in Django, Sockjs server and client, you will get a full-fledged real-time synchronization system for your application.



Update:

At the request of the user timetogo post a link to the resulting management system of task lists: Cloud Checklist

The work of the system can be observed when opening the same sheet of tasks in two separate sessions of the Internet browser. In this scenario, of course, having two monitors is a huge plus.