ZeroRPC is a lightweight, reliable library for distributed communication between servers.

The other day I needed to implement some kind of own statsd-like server for collecting metrics, but with a few narrowly-specific features, under which without a good file, no ready or semi-ready solution could go. In this connection, it was decided to implement a simple client-server protocol in python using tcp / udp soket's. I will make a reservation that I was familiar with network programming, and I remain as far as, although there was a general understanding of the stack tcp / ip. The solution to the forehead on synthetics proved to be wonderful, but as soon as I loaded it with more or less real data (only about 20k messages per second from several streams), it began to show its underwater pebbles. Probably, I just could not properly prepare raw sockets, but the task needed to be solved quickly, there was no time for a deep understanding and the study of network programming, so I started looking for solutions where I would have thought up at least half of them. The search led me to the ZeroRPC library, which was not so long ago, as I understood, released into the world from the depth of dotCloud.

I was surprised that I found only one mention of this development on the Habré, and then in a very casual way, so I decided to write this note.


As the developers of the library itself write on the official website, ZeroRPC is an easy, reliable, language-independent library for distributed communication between servers.
ZeroRPC is built on top of the stack ZeroMQ and MessagePack , which has already been mentioned many times in Habré. And out of the box supports stream responses, heartbeats, timeout detection and even automatic reconnection after some file.
')
It sounded good and I decided to try. By and large, I have not yet studied all the possibilities of the library and will only talk about how I applied it in my case. But, perhaps, it will already interest you to read and, perhaps, try ZeroMQ in your projects.

Installation is simple and straightforward, as with most python libraries:

pip install zerorpc


for herself she will attract another couple of libs, among which are:

• gevent,
• msgpack-python
• pyzmq

It is not put in a couple of seconds, so if the installation suddenly stopped in one place, do not beat the panic - soon everything will continue.

According to the documentation, it is already possible to start using it without writing any code.

zerorpc --server --bind tcp://*:1234 time


I willingly believe, but I was not very interested. The examples on the main page of the site looked more interesting.

Server

 import zerorpc class HelloRPC(object): def hello(self, name): return "Hello, %s" % name s = zerorpc.Server(HelloRPC()) s.bind("tcp://0.0.0.0:4242") s.run() 

Customer

 import zerorpc c = zerorpc.Client() c.connect("tcp://127.0.0.1:4242") print c.hello("RPC") 

Copy-paste, run, works.
Cool, but hello worlds always work, so let's make the primary tests a little more difficult.

Server

 import zerorpc, time class StreamingRPC(object): count = 0 def commit(self, cid): self.count += 1 print(str(self.count)) return str(cid)+'_done' s = zerorpc.Server(StreamingRPC()) s.bind("tcp://0.0.0.0:4242") s.run() 

Customer

 import zerorpc c = zerorpc.Client() c.connect("tcp://127.0.0.1:4242") for i in range(0,290000): print c.commit(i) 

And we will launch several clients in order to see what will happen to the packages.
I was most worried about this issue, because in my implementation I rested against the fact that under heavy load I lost more than 40% of the messages on udp sockets, which is actually not very surprising ...
The test of course showed that everything is not as fast as it was on sockets, but not a single packet loss. A little later, I spent another couple of tests with threads and writing to the database and was quite pleased. After a few days of tests, I rewrote the server and client socket code for my metric, and now, after two weeks, the metric collects a bunch of data and a “single break”. Moreover, having increased the speed of data collection, I did not see a drawdown in performance. At the moment, the metric collects at the peak of up to 100k messages per second and feels great.

Of course, in my case, ZeroRPC is most likely redundant, and I am sure that if I had a more knowledgeable programmer, he would be able to defeat packet loss, hangup sessions, and drop connections of sockets and implement a more productive solution.
But then I would not meet with ZeroRPC and would not tell about it here.
And maybe other features of this library can be very useful in working life reading the developers of this note.

I recommend to see-read the report Jérôme Petazzoni from dotCloud about why and why they did it.
pycon-2012-notes.readthedocs.org/en/latest/dotcloud_zerorpc.html
Project site - www.zerorpc.io
GitHub - github.com/etsy/statsd