Redis, hiredis, libev and multithread. Part 2
In the continuation of the first part I want to tell you how it all really works. A lot of time was put on tests and debugging, and now I want to lay out detailed recommendations on the results of the studies that were conducted.
Attention! Research was conducted not to understand why I need it, but to understand how it works!
')
So, what we had in the first part: pure C code, which was simply included in C ++ code, happily compiled and gave some results, while not very often entering data into the database. But we still want to get closer to real-time, so we do a stress test, and we get ...
The first thing I stumbled upon is falling off the client from the server in 10-15 seconds, which is logical, of course, when data is expected by the server in asynchronous mode, but they do not arrive. To get rid of this misunderstanding, it is necessary to fasten the sending of the “PING” command to the server with a certain periodicity, say, once every 100 μs (one hundred microseconds) after the previous request to the server. This will not affect the operation of the server, and the application will maintain the connection when there is no data.
After screwing up the ping, I began to observe in surprise the fall into the cortex, and in the place where the manual on hiredis recommended cleaning the memory itself, here is a quote: “If it is a reply, it is free. When the callback for a command is non-NULL, it’s responsible for cleaning up the reply. ” However, the reply was not NULL, but with a stable drop in the same freeReplyObject function. Well, what to do, got into the sources of hiredis, and ... The memory is cleaned 2 times! To start, I removed the memory cleaning for ping only, but later it turned out that the Callback function does not need to clean the memory at all. And still it is not necessary to clean it in functions that relate to receiving requests from other threads (those functions that belong to async-watcher), because the transmitted pointers are cleaned in the ev_io_stop function, which is called for each of the io-wacher'ov during remove request from queue. At the same time, of course, with the so-called “private data”, it is necessary to act as Taras Bulba: “I gave birth to you - I will kill you”. By the way, for ping, you can specify private data as NULL, nothing falls anywhere, no memory leaks are observed.
Amazing on the one hand. When I tried to download 10k push requests from a single thread to the database for 10 seconds (the figure is nothing at all at the stated Redis performance of 120k requests per second), I again encountered the problem of rolling back the connection from the DBMS. We climb again into the subsoil, we make a trace and ... We begin to realize the phrase “multithreading is not supported by default, because there is no unique algorithm to make a thread-safe implementation”. What is going on? The following happens: a semaphore that restricts access to the record to the request buffer from other threads is insufficient, because this buffer (and in hiredis it is present and can consume an infinite amount of resources) grows, and data in Redis is not sent, apparently The event loop does not have access to the read buffer while recording is in progress. Initially, I solved the problem by setting a timeout between requests, then I moved the semaphore to the adapter to libev, setting the waiting for the semaphore to add a new request and releasing the semaphore after sending it to the DBMS. However, until the end to deal with the problem of the need for timeout 1mks between requests I have not yet succeeded. Perhaps in the next part (if it will be) I will describe the recipe. Based on all this:
As a result, I managed to achieve ~ 600 push requests per second to Redis with one data preprocessing in the service (it did not count how long it took). In general, this is enough for me to begin with, but I will continue to dig into the increase in the number of threads and even more correct synchronization of adding requests to the buffer.
Whatever one may say, hiredis is a young library, in the process of debugging its code, I more than once climbed into its depths. What is not very happy, so this is the fact that there are often constructions of this approximately the form:
In general, for myself, I patched, wrote to the developers, did not respond quickly, but I think they will fix it.
Thanks for attention. I expect critics and comments.
Attention! Research was conducted not to understand why I need it, but to understand how it works!
')
So, what we had in the first part: pure C code, which was simply included in C ++ code, happily compiled and gave some results, while not very often entering data into the database. But we still want to get closer to real-time, so we do a stress test, and we get ...
Hiredis + Libev: ping me.
The first thing I stumbled upon is falling off the client from the server in 10-15 seconds, which is logical, of course, when data is expected by the server in asynchronous mode, but they do not arrive. To get rid of this misunderstanding, it is necessary to fasten the sending of the “PING” command to the server with a certain periodicity, say, once every 100 μs (one hundred microseconds) after the previous request to the server. This will not affect the operation of the server, and the application will maintain the connection when there is no data.
Hiredis + Libev: be clean, but without fanaticism.
After screwing up the ping, I began to observe in surprise the fall into the cortex, and in the place where the manual on hiredis recommended cleaning the memory itself, here is a quote: “If it is a reply, it is free. When the callback for a command is non-NULL, it’s responsible for cleaning up the reply. ” However, the reply was not NULL, but with a stable drop in the same freeReplyObject function. Well, what to do, got into the sources of hiredis, and ... The memory is cleaned 2 times! To start, I removed the memory cleaning for ping only, but later it turned out that the Callback function does not need to clean the memory at all. And still it is not necessary to clean it in functions that relate to receiving requests from other threads (those functions that belong to async-watcher), because the transmitted pointers are cleaned in the ev_io_stop function, which is called for each of the io-wacher'ov during remove request from queue. At the same time, of course, with the so-called “private data”, it is necessary to act as Taras Bulba: “I gave birth to you - I will kill you”. By the way, for ping, you can specify private data as NULL, nothing falls anywhere, no memory leaks are observed.
Hiredis + Libev + Multithread: I choke!
Amazing on the one hand. When I tried to download 10k push requests from a single thread to the database for 10 seconds (the figure is nothing at all at the stated Redis performance of 120k requests per second), I again encountered the problem of rolling back the connection from the DBMS. We climb again into the subsoil, we make a trace and ... We begin to realize the phrase “multithreading is not supported by default, because there is no unique algorithm to make a thread-safe implementation”. What is going on? The following happens: a semaphore that restricts access to the record to the request buffer from other threads is insufficient, because this buffer (and in hiredis it is present and can consume an infinite amount of resources) grows, and data in Redis is not sent, apparently The event loop does not have access to the read buffer while recording is in progress. Initially, I solved the problem by setting a timeout between requests, then I moved the semaphore to the adapter to libev, setting the waiting for the semaphore to add a new request and releasing the semaphore after sending it to the DBMS. However, until the end to deal with the problem of the need for timeout 1mks between requests I have not yet succeeded. Perhaps in the next part (if it will be) I will describe the recipe. Based on all this:
Hiredis + Libev + Multithread: squeeze the maximum out of one stream.
As a result, I managed to achieve ~ 600 push requests per second to Redis with one data preprocessing in the service (it did not count how long it took). In general, this is enough for me to begin with, but I will continue to dig into the increase in the number of threads and even more correct synchronization of adding requests to the buffer.
Hiredis: before assembly, carefully process the file.
Whatever one may say, hiredis is a young library, in the process of debugging its code, I more than once climbed into its depths. What is not very happy, so this is the fact that there are often constructions of this approximately the form:
<some_struct> *p = (some_struct*)malloc(sizeof(*p)); // , . // char[] - - , , (err), : ... = ... sizeof(err) ...; // , ... In general, for myself, I patched, wrote to the developers, did not respond quickly, but I think they will fix it.
Thanks for attention. I expect critics and comments.
Source: https://habr.com/ru/post/135042/
All Articles