MQ selection for high load project
Modern scalable systems consist of microservices , each of which is responsible for its limited task. This architecture allows you to prevent excessive expansion of the source code and control technical debt.
There are dozens of microservices in our project, each of which is reserved: two or more absolutely identical copies of the service are installed on different physical servers, and the client (the other microservice) can access any of them independently.
If the microservice stops responding as a result of an accident, its clients should be instantly redirected to the backup. To control the flow of requests often use the so-called message queues (message queues).
')
The queue we recently used has ceased to suit us in terms of fault tolerance and we replaced it. Below we share our selection experiences.
It is natural to expect a delivery guarantee from the message queue management service. If the recipient of the message is unavailable, the queue saves the message received from the sender to the disk, and then retries the delivery until a live recipient is found.
The leader in popularity among developers is RabbitMQ . This is a time-tested Enterprise class solution with delivery guarantees, flexible routing and support for all kinds of standards. Project managers love it, as in the early 80s computer buyers loved the IBM PC. This love is most accurately expressed by the phrase “Nobody ever got fired for buying IBM.”
RabbitMQ was not suitable for us because it is slow and expensive to maintain.
RabbitMQ performance does not exceed tens of thousands of messages per second . This is a good result for many applications, but completely unsatisfactory for our case.
Configuring the RabbitMQ cluster is not easy, it takes away valuable devops resources. In addition, we had a glimpse of complaints about the work of the cluster - it does not know how to merge queues with conflicts that have arisen in the “split brain” situation (when two isolated nodes are formed due to a network break, each of which considers it to be the main one).
We looked at Apache Kafka , which was born inside LinkedIn as a log aggregation system. Kafka can squeeze more performance out of the disk subsystem than RabbitMQ, because it writes data sequentially (sequential I / O) and not randomly (random I / O). But no guarantees that the recording on the disc will always occur consistently can not be obtained.
In Kafka, data is divided into sections (partition), and in order to comply with the order of delivery, each message recipient reads data from exactly one section. This can lead to a queue lock in the event that the recipient, for whatever reason, processes the message more slowly than usual.
In addition, a separate service (zookeeper) is required to manage the Kafka cluster, which again complicates maintenance and loads devops.
We are not ready to risk loss of productivity in production, so we continued our search.
There is a great sign from Jeff Dean, a Google veteran (has been working there since 1999):
It can be seen that writing to disk is 15 times slower than sending over the network.
Paradoxically: your addressee is online, transfer the message there 15 times faster than writing to disk, but for some reason you write it to disk.
“Understandably, this is necessary to guarantee delivery,” you will say. “After all, if the addressee receives a message, but, not having time to process it, it will fall due to iron failure, the queue must deliver it again.”
This is true, but there is no guarantee. After all, if the sender falls at the time of sending the message or the queue process itself drops before writing to disk, the message will disappear. It turns out that the queue only creates the illusion of a delivery guarantee, and messages can still be lost.
In order not to lose messages within the queue process, you can simply ... remove the queue process and replace it with the library built into the microservice process.
Many developers are familiar with the ZeroMQ library. It shows fantastic speed, digesting millions of messages per second. However, it (for ideological reasons) does not have built-in monitoring and cluster management, so when using it, the load on devops is even higher. We continued to look for more practical options.
At some point we were almost desperate and it seemed to us that it would be easier to write the queue by ourselves on top of the DBMS. This may be a SQL database or one of numerous NoSQL solutions.
For example, Redis has special functions for implementing queues. Since Redis keeps the data in memory, the performance is excellent. This option was reasonable, but embarrassed that the Sentinel add-on, designed to merge several Redis nodes into a cluster, looked somewhat artificial, as if attached to the side.
When using a classical DBMS, you would have to use the “long polling” technique to receive messages. It is ugly and fraught with delays in delivery. Yes, and did not want to write on my knee.
Intuition suggested that we were not the first to look for a queue with reasonable requirements for performance and ease of administration, and in 2017 our task should have a ready-made solution.
NATS is a relatively young project created by Derek Collison, with more than 20 years of experience working on distributed message queues.
We were impressed by the simplicity of administering the NATS cluster. To connect a new node, the NATS process only needs to specify the address of any other node in the cluster, and it instantly downloads the entire topology and determines the live / dead nodes. Messages in NATS are grouped by topic, and each node knows which nodes have live subscribers to which topics. All messages in the cluster are delivered directly from the sender to the recipient, without intermediate steps and with minimal delay.
By performance, NATS is ahead of all lines with “guaranteed delivery” . NATS is written in Go, but has client libraries for all popular languages. In addition, NATS clients also know the cluster topology and are able to reconnect themselves in case of loss of communication with their site.
Since NATS does not write messages to disk, recipient services must carefully shut down their work — first unsubscribe from new messages, then process received messages and then stop the process.
Sender services should retry sending messages in case of errors. (However, this is not specific to NATS and this is what needs to be done when working with any queue).
In order to know for sure about the lost messages, we made the simplest logging: we write down the time of sending and the time of receiving.
We installed the NATS process on all virtual machines with microservices. According to the results of observation for 2 months, NATS did not lose a single message.
We are happy with our choice. We hope our experience will be useful to you.
There are dozens of microservices in our project, each of which is reserved: two or more absolutely identical copies of the service are installed on different physical servers, and the client (the other microservice) can access any of them independently.
If the microservice stops responding as a result of an accident, its clients should be instantly redirected to the backup. To control the flow of requests often use the so-called message queues (message queues).
')
The queue we recently used has ceased to suit us in terms of fault tolerance and we replaced it. Below we share our selection experiences.
Proven solution
It is natural to expect a delivery guarantee from the message queue management service. If the recipient of the message is unavailable, the queue saves the message received from the sender to the disk, and then retries the delivery until a live recipient is found.
The leader in popularity among developers is RabbitMQ . This is a time-tested Enterprise class solution with delivery guarantees, flexible routing and support for all kinds of standards. Project managers love it, as in the early 80s computer buyers loved the IBM PC. This love is most accurately expressed by the phrase “Nobody ever got fired for buying IBM.”
RabbitMQ was not suitable for us because it is slow and expensive to maintain.
RabbitMQ performance does not exceed tens of thousands of messages per second . This is a good result for many applications, but completely unsatisfactory for our case.
Configuring the RabbitMQ cluster is not easy, it takes away valuable devops resources. In addition, we had a glimpse of complaints about the work of the cluster - it does not know how to merge queues with conflicts that have arisen in the “split brain” situation (when two isolated nodes are formed due to a network break, each of which considers it to be the main one).
Queue based on distributed log
We looked at Apache Kafka , which was born inside LinkedIn as a log aggregation system. Kafka can squeeze more performance out of the disk subsystem than RabbitMQ, because it writes data sequentially (sequential I / O) and not randomly (random I / O). But no guarantees that the recording on the disc will always occur consistently can not be obtained.
In Kafka, data is divided into sections (partition), and in order to comply with the order of delivery, each message recipient reads data from exactly one section. This can lead to a queue lock in the event that the recipient, for whatever reason, processes the message more slowly than usual.
In addition, a separate service (zookeeper) is required to manage the Kafka cluster, which again complicates maintenance and loads devops.
We are not ready to risk loss of productivity in production, so we continued our search.
"Guaranteed" message delivery
There is a great sign from Jeff Dean, a Google veteran (has been working there since 1999):
Latency Comparison Numbers -------------------------- L1 cache reference 0.5 ns Branch mispredict 5 ns L2 cache reference 7 ns 14x L1 cache Mutex lock/unlock 25 ns Main memory reference 100 ns 20x L2 cache, 200x L1 cache Compress 1K bytes with Zippy 3,000 ns 3 us Send 1K bytes over 1 Gbps network 10,000 ns 10 us Read 4K randomly from SSD* 150,000 ns 150 us ~1GB/sec SSD Read 1 MB sequentially from memory 250,000 ns 250 us Round trip within same datacenter 500,000 ns 500 us Read 1 MB sequentially from SSD* 1,000,000 ns 1,000 us 1 ms ~1GB/sec SSD, 4X memory Disk seek 10,000,000 ns 10,000 us 10 ms 20x datacenter roundtrip Read 1 MB sequentially from disk 20,000,000 ns 20,000 us 20 ms 80x memory, 20X SSD Send packet CA->Netherlands->CA 150,000,000 ns 150,000 us 150 ms It can be seen that writing to disk is 15 times slower than sending over the network.
Paradoxically: your addressee is online, transfer the message there 15 times faster than writing to disk, but for some reason you write it to disk.
“Understandably, this is necessary to guarantee delivery,” you will say. “After all, if the addressee receives a message, but, not having time to process it, it will fall due to iron failure, the queue must deliver it again.”
This is true, but there is no guarantee. After all, if the sender falls at the time of sending the message or the queue process itself drops before writing to disk, the message will disappear. It turns out that the queue only creates the illusion of a delivery guarantee, and messages can still be lost.
High-performance queues
In order not to lose messages within the queue process, you can simply ... remove the queue process and replace it with the library built into the microservice process.
Many developers are familiar with the ZeroMQ library. It shows fantastic speed, digesting millions of messages per second. However, it (for ideological reasons) does not have built-in monitoring and cluster management, so when using it, the load on devops is even higher. We continued to look for more practical options.
Queue on a DBMS?
At some point we were almost desperate and it seemed to us that it would be easier to write the queue by ourselves on top of the DBMS. This may be a SQL database or one of numerous NoSQL solutions.
For example, Redis has special functions for implementing queues. Since Redis keeps the data in memory, the performance is excellent. This option was reasonable, but embarrassed that the Sentinel add-on, designed to merge several Redis nodes into a cluster, looked somewhat artificial, as if attached to the side.
When using a classical DBMS, you would have to use the “long polling” technique to receive messages. It is ugly and fraught with delays in delivery. Yes, and did not want to write on my knee.
Intuition suggested that we were not the first to look for a queue with reasonable requirements for performance and ease of administration, and in 2017 our task should have a ready-made solution.
Solution found: NATS
NATS is a relatively young project created by Derek Collison, with more than 20 years of experience working on distributed message queues.
We were impressed by the simplicity of administering the NATS cluster. To connect a new node, the NATS process only needs to specify the address of any other node in the cluster, and it instantly downloads the entire topology and determines the live / dead nodes. Messages in NATS are grouped by topic, and each node knows which nodes have live subscribers to which topics. All messages in the cluster are delivered directly from the sender to the recipient, without intermediate steps and with minimal delay.
By performance, NATS is ahead of all lines with “guaranteed delivery” . NATS is written in Go, but has client libraries for all popular languages. In addition, NATS clients also know the cluster topology and are able to reconnect themselves in case of loss of communication with their site.
Results of use in production
Since NATS does not write messages to disk, recipient services must carefully shut down their work — first unsubscribe from new messages, then process received messages and then stop the process.
Sender services should retry sending messages in case of errors. (However, this is not specific to NATS and this is what needs to be done when working with any queue).
In order to know for sure about the lost messages, we made the simplest logging: we write down the time of sending and the time of receiving.
We installed the NATS process on all virtual machines with microservices. According to the results of observation for 2 months, NATS did not lose a single message.
We are happy with our choice. We hope our experience will be useful to you.
Source: https://habr.com/ru/post/326880/
All Articles