IoT cloud on Netty or 10k rec-sec per core
Hello. This post is about a server solution for the Internet of things, which I wrote on asynchronous sockets using the well-known Netty. I will talk about the task we set for ourselves, why I chose Netty, why it doesn’t have alternatives, what Netty’s disadvantages and advantages are and how to get the most out of it. Now our server processes an average of 1.5 billion messages per month and the load is growing by 20% every month. To attract attention - the load on one production server with 4 cores Xeon® CPU E5-2630L v2 @ 2.40GHz with a load of 500 rec-sec.
So let's go.
')
It all started about 2 years ago, when they gave me an arduino. I always wanted to do some interesting device with my own hands. But all these soldering irons, resistors, volts-amps constantly scared me. So it was, until arduino appeared. With Arduina, I was finally able to control the electronics. To say that it was very cool to say nothing. I was happy. But, as is often the case, after mastering the basic skills in microcontrollers, I wanted more - to control devices via the Internet from the phone. Fast googling showed (it was 2 years ago) that currently there is not a single solution that would solve this problem. Not counting the IoT cloud with HTTP API, which was not very convenient to use.
Fortunately, I was not alone. Quite by chance, in my work, I met people who were concerned about the same problems. So our project appeared.
Remotely, the task was clear - control Arduina via the Internet from a mobile or tablet. Well, for example, to be able to change the color of the skirt below.
At that stage, there were no more obvious and clear requirements. After a brief discussion and analysis, we outlined several important points that the project architecture should provide:
I do not consider myself a geek. But already then I had 2 arduinks, 2 phones and a tablet, which I would like to use for control. A small survey among acquaintances showed that the average hardware vendor has 3-4 microcontrollers. It became quite obvious that one potential user of our system can easily open 5-10 connections from different devices. That is, the server must simultaneously handle thousands of parallel connections, even for several hundred users.
Despite its little experience with microcontrollers, it was obvious that the server solution should support 2 client operation modes - pushing and polling. That is, the mobile client must be able to request a status from the piece of iron and, on its part, the piece of hardware should be able to send a message at any time and the mobile client to receive it.
It was difficult to choose a protocol. I really wanted to go the old trodden path - HTTP. But HTTP has several architectural problems. The minimum possible packet is 26 bytes (excluding TCP / IP headers), but in real life the minimum packet will be at least 50 bytes, and considering the weak capabilities of microcontrollers (for Arduino UNO, for example, 2kb RAM and 16 MHz processor) and a short battery life seemed superfluous. Well, if with http poling, everything is fine, then web-scokets are needed with pushing, and this is essentially good old TCP / IP. Among hardwarders, the MQTT protocol is very popular. But I also refused it. He seemed too sophisticated to me. Yes, it was thought out for all occasions. But then we did not need it. Therefore, the server had to support at least several protocols so that we could switch to the necessary one if our choice were not correct.
Since we initially did the project for ourselves, one of the main conditions was the simplicity of setup and deployment. So that even a completely new person, without any skills, could easily and within a few minutes begin to manage his piece of iron and, if necessary, could just as easily and easily deploy a server.
A little thought, it was decided to use asynchronous sockets. I was guided by 2 factors. We need to handle many parallel connections, which blocking sockets do not do very well due to the large stack (256kb) in Java and the cost of context switching between threads. Business logic will be a bit. After all, you just need to send a message from one socket to another (well, who does not make mistakes). All the server’s work essentially comes down to “take a message from socket A” and “send a message to socket B” + a little business logic from above.
A quick analysis of existing solutions revealed the following:
I've been working with netty for 2 years now. And I have something to say. I'll start with the cons.
There is an excellent getting started and everything seems to be fine. But as soon as you begin to take a step to the left, a step to the right, dozens of questions immediately arise. There are no answers to which in the documentation and answers to which are scattered throughout the Internet. Now the situation is much better than it was 2 years ago. But the problem still remains.
As an asynchronous framework, Netty makes it easy to work with asynchronous sockets. Still, with a swoop immediately go to the business logic does not work. It is necessary to study, understand and rethink many things. Perhaps in my case it was the result of many years of work with the blocking API. I do not know. But I still learn something new about Netty.
Often, when working with netties, there is a need to look inside the ready handlers to understand how they work. But the rather large and complex inheritance hierarchy greatly complicates the understanding of the code. I don’t know about you, but if the class has a hierarchy of more than 3 levels of nesting, then my stack that is not full is full. Netty is all saturated with this complex hierarchy. In addition to everything, netty has its own separate package with 4 different types of buffers that netty uses.
At first, it's very easy to shoot yourself with a netty. A small example:
It turns out in the 4th netty (I started with the 3rd) by default, direct buffers are used, so you need to carefully monitor the buffers. The problem of leaks in netty is generally very serious. As a confirmation, Netty has a solution that allows you to track leaks inside the framework. Although if you use ready handlers (that is, you use ready-made HTTP, MQTT, SPDY, ...) protocols, then you will not encounter this.
Due to its popularity, various kinds of bugs are often found in Netty. Now on gitkhab open about 280 tickets. I personally ran into 2 that had to start. Fortunately, with fixes everything happens very quickly.
Now for the good.
With Netty, you have absolutely complete control over the network stack. And that's cool. Because you have the opportunity to work in the abstractions of the framework while avoiding the entire complexity of direct work with sockets. In addition, netty has a bunch of features out of the box that no one offers on the market, for example, the ability to include TCP_FASTOPEN, TCP_MD5SIG, TCP_CORK, SPLICE, support for native Epoll transport, OpenSSL, and a bunch of other things. And it all works out of the box!
TCP, UDP, UDT, SCTP? Easy. HTTP, HTTPS, HTTP / 2, SPDY, Memcached, MQTT? Has already. Connect any of these protocols for a few tens of minutes. And all this has long been written, optimized and tested. An interesting point with HTTP / 2. He was added to the netty about 3 months ago. In nginx for example, its support appeared literally 2 weeks ago. I think it says a lot.
Need to add SSL? Please - SslHandler. Need to compress all traffic? Choose any - JdkZlibEncoder, ZlibEncoder, SnappyFramedEncoder. Need to parse Jason on the go? - JsonObjectDecoder. Need to close inactive sockets? Please - ReadTimeoutHandler. Do you need to completely subtract data from a socket and only then transfer control to the business logic? - well, you understand ...
100k rek-s per core for simplest HTTP GET? With netty, this is true .
Mainly due to the fact that it is used in monsters like Facebook, Twitter, so much attention is paid to optimizations in netty. Another important factor why netty is fast is the use of direct buffers, which greatly reduces the load on the GC. In this case, the complexity of managing these buffers takes on netty.
One of the reasons why netty is fast is its asynchronous nature, which means that there are virtually no locks inside IO handlers (workers) and no loss when switching context between threads. The very nature of Netty forces you to write code that must be thread-safe. Of course, you can hardly avoid synchronization completely, but below I will tell you how to avoid this as much as possible.
To get a ticket and get an answer within an hour is the norm for Netty. This is very cool and helps a lot in development. Especially against the background of insufficient documentation. In general, discussing any question about architecture, performance, or small refactoring is not at all a problem for developers.
To be continued…
So let's go.
')
It all started about 2 years ago, when they gave me an arduino. I always wanted to do some interesting device with my own hands. But all these soldering irons, resistors, volts-amps constantly scared me. So it was, until arduino appeared. With Arduina, I was finally able to control the electronics. To say that it was very cool to say nothing. I was happy. But, as is often the case, after mastering the basic skills in microcontrollers, I wanted more - to control devices via the Internet from the phone. Fast googling showed (it was 2 years ago) that currently there is not a single solution that would solve this problem. Not counting the IoT cloud with HTTP API, which was not very convenient to use.
Fortunately, I was not alone. Quite by chance, in my work, I met people who were concerned about the same problems. So our project appeared.
Problem
Remotely, the task was clear - control Arduina via the Internet from a mobile or tablet. Well, for example, to be able to change the color of the skirt below.
At that stage, there were no more obvious and clear requirements. After a brief discussion and analysis, we outlined several important points that the project architecture should provide:
Many parallel connections
I do not consider myself a geek. But already then I had 2 arduinks, 2 phones and a tablet, which I would like to use for control. A small survey among acquaintances showed that the average hardware vendor has 3-4 microcontrollers. It became quite obvious that one potential user of our system can easily open 5-10 connections from different devices. That is, the server must simultaneously handle thousands of parallel connections, even for several hundred users.
Pushing / polling
Despite its little experience with microcontrollers, it was obvious that the server solution should support 2 client operation modes - pushing and polling. That is, the mobile client must be able to request a status from the piece of iron and, on its part, the piece of hardware should be able to send a message at any time and the mobile client to receive it.
Multi-protocol support
It was difficult to choose a protocol. I really wanted to go the old trodden path - HTTP. But HTTP has several architectural problems. The minimum possible packet is 26 bytes (excluding TCP / IP headers), but in real life the minimum packet will be at least 50 bytes, and considering the weak capabilities of microcontrollers (for Arduino UNO, for example, 2kb RAM and 16 MHz processor) and a short battery life seemed superfluous. Well, if with http poling, everything is fine, then web-scokets are needed with pushing, and this is essentially good old TCP / IP. Among hardwarders, the MQTT protocol is very popular. But I also refused it. He seemed too sophisticated to me. Yes, it was thought out for all occasions. But then we did not need it. Therefore, the server had to support at least several protocols so that we could switch to the necessary one if our choice were not correct.
Simplicity
Since we initially did the project for ourselves, one of the main conditions was the simplicity of setup and deployment. So that even a completely new person, without any skills, could easily and within a few minutes begin to manage his piece of iron and, if necessary, could just as easily and easily deploy a server.
Choosing a solution
A little thought, it was decided to use asynchronous sockets. I was guided by 2 factors. We need to handle many parallel connections, which blocking sockets do not do very well due to the large stack (256kb) in Java and the cost of context switching between threads. Business logic will be a bit. After all, you just need to send a message from one socket to another (well, who does not make mistakes). All the server’s work essentially comes down to “take a message from socket A” and “send a message to socket B” + a little business logic from above.
What is on the market?
A quick analysis of existing solutions revealed the following:
- Apache MINA is a dead project that has not been developed for a year already 4.
- Grizzly - I dropped it as I did not find the projects that use it.
- Xnio - rejected because of the 60 stars on the githaba. And those, apparently the developers.
- Vert.x - at that time was positioned as a direct competitor to Netty. And then he had either a closed or a paid core. It was 2 years ago and I already forgot the details. Now the situation has changed. So maybe it makes sense to look again.
- Akka is a very interesting thing. And quite a good solution, I preferred netty for a simple reason - Akka can be built on netty, but not on Akke netty. Well, the very fact that the distributed actors in Akka use netty, as if it hints. Well, netty uses thousands of projects around the world including facebook, linkedin and twitter.
- Netty - the choice is obvious
Disadvantages of netty
I've been working with netty for 2 years now. And I have something to say. I'll start with the cons.
Bad documentation
There is an excellent getting started and everything seems to be fine. But as soon as you begin to take a step to the left, a step to the right, dozens of questions immediately arise. There are no answers to which in the documentation and answers to which are scattered throughout the Internet. Now the situation is much better than it was 2 years ago. But the problem still remains.
Complexity
As an asynchronous framework, Netty makes it easy to work with asynchronous sockets. Still, with a swoop immediately go to the business logic does not work. It is necessary to study, understand and rethink many things. Perhaps in my case it was the result of many years of work with the blocking API. I do not know. But I still learn something new about Netty.
Too many abstractions
Often, when working with netties, there is a need to look inside the ready handlers to understand how they work. But the rather large and complex inheritance hierarchy greatly complicates the understanding of the code. I don’t know about you, but if the class has a hierarchy of more than 3 levels of nesting, then my stack that is not full is full. Netty is all saturated with this complex hierarchy. In addition to everything, netty has its own separate package with 4 different types of buffers that netty uses.
Easy to shoot in the leg
At first, it's very easy to shoot yourself with a netty. A small example:
ByteBuf in ... in.readBytes(length); // ByteBuf in ... in.readSlice(length); // It turns out in the 4th netty (I started with the 3rd) by default, direct buffers are used, so you need to carefully monitor the buffers. The problem of leaks in netty is generally very serious. As a confirmation, Netty has a solution that allows you to track leaks inside the framework. Although if you use ready handlers (that is, you use ready-made HTTP, MQTT, SPDY, ...) protocols, then you will not encounter this.
There are still bugs
Due to its popularity, various kinds of bugs are often found in Netty. Now on gitkhab open about 280 tickets. I personally ran into 2 that had to start. Fortunately, with fixes everything happens very quickly.
Netty Pros
Now for the good.
Full control
With Netty, you have absolutely complete control over the network stack. And that's cool. Because you have the opportunity to work in the abstractions of the framework while avoiding the entire complexity of direct work with sockets. In addition, netty has a bunch of features out of the box that no one offers on the market, for example, the ability to include TCP_FASTOPEN, TCP_MD5SIG, TCP_CORK, SPLICE, support for native Epoll transport, OpenSSL, and a bunch of other things. And it all works out of the box!
Multi-protocol support
TCP, UDP, UDT, SCTP? Easy. HTTP, HTTPS, HTTP / 2, SPDY, Memcached, MQTT? Has already. Connect any of these protocols for a few tens of minutes. And all this has long been written, optimized and tested. An interesting point with HTTP / 2. He was added to the netty about 3 months ago. In nginx for example, its support appeared literally 2 weeks ago. I think it says a lot.
Many ready handlers
Need to add SSL? Please - SslHandler. Need to compress all traffic? Choose any - JdkZlibEncoder, ZlibEncoder, SnappyFramedEncoder. Need to parse Jason on the go? - JsonObjectDecoder. Need to close inactive sockets? Please - ReadTimeoutHandler. Do you need to completely subtract data from a socket and only then transfer control to the business logic? - well, you understand ...
Fast
100k rek-s per core for simplest HTTP GET? With netty, this is true .
Mainly due to the fact that it is used in monsters like Facebook, Twitter, so much attention is paid to optimizations in netty. Another important factor why netty is fast is the use of direct buffers, which greatly reduces the load on the GC. In this case, the complexity of managing these buffers takes on netty.
No synchronize
One of the reasons why netty is fast is its asynchronous nature, which means that there are virtually no locks inside IO handlers (workers) and no loss when switching context between threads. The very nature of Netty forces you to write code that must be thread-safe. Of course, you can hardly avoid synchronization completely, but below I will tell you how to avoid this as much as possible.
Great community
To get a ticket and get an answer within an hour is the norm for Netty. This is very cool and helps a lot in development. Especially against the background of insufficient documentation. In general, discussing any question about architecture, performance, or small refactoring is not at all a problem for developers.
To be continued…
Source: https://habr.com/ru/post/268895/
All Articles