How we injected cache on tarantool db
Good day!
I want to share with you a story about the implementation of cache on the Tarantool database and its features of work.
I work as a Java developer in a telecommunications company. The main task: the implementation of business logic for the platform that the company bought from the vendor. Of the first features, this is the work on soap and the almost complete lack of caching, except in JVM memory. All this is of course good as long as the number of copies of the application does not exceed two dozen ...
In the course of the work and the emergence of an understanding of the features of the platform, an attempt was made to make caching. At that time, MongoDB was already launched and, as a result, we did not receive any specific positive results as in the test.
')
Upon further search for alternatives and the advice of my good friend mr_elzor , it was decided to try the Tarantool database.
In a cursory study, only lua appeared in doubt, since I had not written on it before from the word “absolutely”. But pushing all doubts aside, proceeded to install. About the closed networks and firewalls I think very few people are interested, but I advise you to try to get around them and put everything from public sources.
Test servers with configuration: 8 Cpu, 16 GB Ram, 100 Gb HDD, Debian 9.4.
The installation was according to the instructions from the site. And so I got an example option. The thought of a visual interface appeared immediately, with which it would be convenient to work with support. In a quick search, I found and configured tarantool-admin . Works in Docker and covers support tasks by 100%, at least for now.
But let's talk about more interesting.
The next thought was to set up your own version in the master-slave configuration within the same server, since the documentation contains only examples with two different servers.
Having spent some time understanding lua and a description of the configuration and launching the wizard.
I immediately fall into a stupor and do not understand why the error, but I see that it is in the status of "loading".
I start slave:
And I see the same error. This is where I start to strain at all and don’t understand what is happening, because there’s nothing in the documentation about this at all ... But when checking the status, I see that it didn’t start at all, although it says that the status is “running”:
But at the same time, the master began to work:
Restarting the slave does not help. I wonder why?
I stop the master. And perform the actions in reverse order.
I see that the slave is trying to start.
I start the master and see that he did not rise and in general passed to the orphan status, and the slave fell altogether.
It becomes even more interesting.
In the logs on the slave I see that he even saw the master and tried to synchronize.
And the attempt was successful:
He even started:
But for unknown reasons, he lost the connection and fell:
Attempting to start the slave again does not help.
Now I delete the files created by the instances. In my case, I delete everything from the / var / lib / tarantool directory.
I start the slave first, and only then master. And lo and behold ...
I did not find any explanation for this behavior, except as a "feature of this software."
This situation will appear every time if your server has completely rebooted.
Upon further analysis of the architecture of this software, it turns out that it is envisaged to use only one vCPU for one instance and many more resources remain free.
In ideology from n vCPU, we can raise the master and n-2 slaves for reading.
Considering that on a test server of 8 vCPU we can raise a master and 6 instances for reading.
I copy the file for the slave, correct the ports and start, i.e. added a few more slaves.
Important! When adding another instance, you must register it on the master.
But you must first start the new slave, and only then restart the wizard.
I already had a configuration running with a master and two slaves.
I decided to add a third slave.
I registered it on the master and restarted the master first, and this is what I saw:
Those. our master became a loner, and replication collapsed.
Running a new slave will no longer help and will end in an error:
And in the logs I saw a record of little information:
Stop the wizard and start a new slave. There will also be an error, as on the first launch, but we will see that it has loading status.
But when you start the master, the new slave drops, and the master does not go with the status of “running”.
In this situation, the only way out. As I wrote earlier, I delete the files created by the instances and run slaves first, and then master.
Everything started successfully.
That's how trial and error figured out how to configure and run replication correctly.
As a result, the following configuration was collected:
2 servers.
2 master. Hot reserve.
12 slaves. All active.
In the logic of tarantool, http.server was used so as not to fence an additional adapter (remember about the vendor, platform and soap) or to attach a library to each business process.
In order to avoid discrepancies between the masters, on the balancer (NetScaler, HAProxy or any other your favorite) we prescribe the rule of reserve, i.e. insert, update, delete operations go only to the first active master.
At this time, the second simply replicates the records from the first. The slaves themselves connect to the first specified master from the configuration, which is what we need in this situation.
On lua implemented CRUD operations for key-value. At the moment, this is enough to solve the problem.
In view of the peculiarities of working with soap, a proxying business process was implemented, in which the logic of working with a tarantula via http was laid.
If the key data is present, then it is returned immediately. If not, then the request goes to the master system, and is saved in the Tarantool database.
As a result, one business process in tests handles up to 4k requests. At the same time response from the tarantula ~ 1ms. Average response time up to 3ms.
Here is some information from the tests:
There were 50 business processes that go to 4 master systems and cache data in their memory. Duplication of information in full growth on each instance. Considering that java loves memory anyway ... perspective is not the best.
50 business processes request information through the cache. Now information from the 4 instances of the wizard is stored in one place, and is not cached in memory on each instance. It was possible to significantly reduce the load on the master system, there are no duplicates of information and the memory consumption on the instances with business logic has decreased.
An example of the amount of information stored in the memory of a tarantula:
At the end of the day, these figures may double, but there is no “drawdown” in performance.
In combat, the current version creates 2k - 2.5k requests per second of the real load. Average response time is similar to tests up to 3ms.
If you look at htop on one of the servers with tarantool, then we will see that they “cool off”:
Despite all the subtleties and nuances of the Tarantool database, you can achieve great performance.
I hope this project will be developed and these uncomfortable moments will be resolved.
I want to share with you a story about the implementation of cache on the Tarantool database and its features of work.
I work as a Java developer in a telecommunications company. The main task: the implementation of business logic for the platform that the company bought from the vendor. Of the first features, this is the work on soap and the almost complete lack of caching, except in JVM memory. All this is of course good as long as the number of copies of the application does not exceed two dozen ...
In the course of the work and the emergence of an understanding of the features of the platform, an attempt was made to make caching. At that time, MongoDB was already launched and, as a result, we did not receive any specific positive results as in the test.
')
Upon further search for alternatives and the advice of my good friend mr_elzor , it was decided to try the Tarantool database.
In a cursory study, only lua appeared in doubt, since I had not written on it before from the word “absolutely”. But pushing all doubts aside, proceeded to install. About the closed networks and firewalls I think very few people are interested, but I advise you to try to get around them and put everything from public sources.
Test servers with configuration: 8 Cpu, 16 GB Ram, 100 Gb HDD, Debian 9.4.
The installation was according to the instructions from the site. And so I got an example option. The thought of a visual interface appeared immediately, with which it would be convenient to work with support. In a quick search, I found and configured tarantool-admin . Works in Docker and covers support tasks by 100%, at least for now.
But let's talk about more interesting.
The next thought was to set up your own version in the master-slave configuration within the same server, since the documentation contains only examples with two different servers.
Having spent some time understanding lua and a description of the configuration and launching the wizard.
# systemctl start tarantool@master Job for tarantool@master.service failed because the control process exited with error code. See "systemctl status tarantool@master.service" and "journalctl -xe" for details. I immediately fall into a stupor and do not understand why the error, but I see that it is in the status of "loading".
# systemctl status tarantool@master ● tarantool@master.service - Tarantool Database Server Loaded: loaded (/lib/systemd/system/tarantool@.service; enabled; vendor preset: enabled) Active: activating (start) since Tue 2019-02-19 17:03:24 MSK; 17s ago Docs: man:tarantool(1) Process: 20111 ExecStop=/usr/bin/tarantoolctl stop master (code=exited, status=0/SUCCESS) Main PID: 20120 (tarantool) Status: "loading" Tasks: 5 (limit: 4915) CGroup: /system.slice/system-tarantool.slice/tarantool@master.service └─20120 tarantool master.lua <loading> Feb 19 17:03:24 tarantuldb-tst4 systemd[1]: Starting Tarantool Database Server... Feb 19 17:03:24 tarantuldb-tst4 tarantoolctl[20120]: Starting instance master... Feb 19 17:03:24 tarantuldb-tst4 tarantoolctl[20120]: Run console at unix/:/var/run/tarantool/master.control Feb 19 17:03:24 tarantuldb-tst4 tarantoolctl[20120]: started I start slave:
# systemctl start tarantool@slave2 Job for tarantool@slave2.service failed because the control process exited with error code. See "systemctl status tarantool@slave2.service" and "journalctl -xe" for details. And I see the same error. This is where I start to strain at all and don’t understand what is happening, because there’s nothing in the documentation about this at all ... But when checking the status, I see that it didn’t start at all, although it says that the status is “running”:
# systemctl status tarantool@slave2 â—Ź tarantool@slave2.service - Tarantool Database Server Loaded: loaded (/lib/systemd/system/tarantool@.service; enabled; vendor preset: enabled) Active: failed (Result: exit-code) since Tue 2019-02-19 17:04:52 MSK; 27s ago Docs: man:tarantool(1) Process: 20258 ExecStop=/usr/bin/tarantoolctl stop slave2 (code=exited, status=0/SUCCESS) Process: 20247 ExecStart=/usr/bin/tarantoolctl start slave2 (code=exited, status=1/FAILURE) Main PID: 20247 (code=exited, status=1/FAILURE) Status: "running" Feb 19 17:04:52 tarantuldb-tst4 systemd[1]: tarantool@slave2.service: Unit entered failed state. Feb 19 17:04:52 tarantuldb-tst4 systemd[1]: tarantool@slave2.service: Failed with result 'exit-code'. Feb 19 17:04:52 tarantuldb-tst4 systemd[1]: tarantool@slave2.service: Service hold-off time over, scheduling restart. Feb 19 17:04:52 tarantuldb-tst4 systemd[1]: Stopped Tarantool Database Server. Feb 19 17:04:52 tarantuldb-tst4 systemd[1]: tarantool@slave2.service: Start request repeated too quickly. Feb 19 17:04:52 tarantuldb-tst4 systemd[1]: Failed to start Tarantool Database Server. Feb 19 17:04:52 tarantuldb-tst4 systemd[1]: tarantool@slave2.service: Unit entered failed state. Feb 19 17:04:52 tarantuldb-tst4 systemd[1]: tarantool@slave2.service: Failed with result 'exit-code'. But at the same time, the master began to work:
# ps -ef | grep taran taranto+ 20158 1 0 17:04 ? 00:00:00 tarantool master.lua <running> root 20268 2921 0 17:06 pts/1 00:00:00 grep taran Restarting the slave does not help. I wonder why?
I stop the master. And perform the actions in reverse order.
I see that the slave is trying to start.
# ps -ef | grep taran taranto+ 20399 1 0 17:09 ? 00:00:00 tarantool slave2.lua <loading> I start the master and see that he did not rise and in general passed to the orphan status, and the slave fell altogether.
# ps -ef | grep taran taranto+ 20428 1 0 17:09 ? 00:00:00 tarantool master.lua <orphan> It becomes even more interesting.
In the logs on the slave I see that he even saw the master and tried to synchronize.
2019-02-19 17:13:45.113 [20751] iproto/101/main D> binary: binding to 0.0.0.0:3302... 2019-02-19 17:13:45.113 [20751] iproto/101/main I> binary: bound to 0.0.0.0:3302 2019-02-19 17:13:45.113 [20751] iproto/101/main D> binary: listening on 0.0.0.0:3302... 2019-02-19 17:13:45.113 [20751] iproto D> cpipe_flush_cb: locking &endpoint->mutex 2019-02-19 17:13:45.113 [20751] iproto D> cpipe_flush_cb: unlocking &endpoint->mutex 2019-02-19 17:13:45.113 [20751] main D> cbus_endpoint_fetch: locking &endpoint->mutex 2019-02-19 17:13:45.113 [20751] main D> cbus_endpoint_fetch: unlocking &endpoint->mutex 2019-02-19 17:13:45.113 [20751] main/101/slave2 I> connecting to 1 replicas 2019-02-19 17:13:45.113 [20751] main/106/applier/replicator@tarantuldb-t D> => CONNECT 2019-02-19 17:13:45.114 [20751] main/106/applier/replicator@tarantuldb-t I> remote master 825af7c3-f8df-4db0-8559-a866b8310077 at 10.78.221.74:3301 running Tarantool 1.10.2 2019-02-19 17:13:45.114 [20751] main/106/applier/replicator@tarantuldb-t D> => CONNECTED 2019-02-19 17:13:45.114 [20751] main/101/slave2 I> connected to 1 replicas 2019-02-19 17:13:45.114 [20751] coio V> loading vylog 14 2019-02-19 17:13:45.114 [20751] coio V> done loading vylog 2019-02-19 17:13:45.114 [20751] main/101/slave2 I> recovery start 2019-02-19 17:13:45.114 [20751] main/101/slave2 I> recovering from `/var/lib/tarantool/cache_slave2/00000000000000000014.snap' 2019-02-19 17:13:45.114 [20751] main/101/slave2 D> memtx_tuple_new(47) = 0x7f99a4000080 2019-02-19 17:13:45.114 [20751] main/101/slave2 I> cluster uuid 4035b563-67f8-4e85-95cc-e03429f1fa4d 2019-02-19 17:13:45.114 [20751] main/101/slave2 D> memtx_tuple_new(11) = 0x7f99a4004080 2019-02-19 17:13:45.114 [20751] main/101/slave2 D> memtx_tuple_new(17) = 0x7f99a4008068 And the attempt was successful:
2019-02-19 17:13:45.118 [20751] main/101/slave2 D> memtx_tuple_new(40) = 0x7f99a40004c0 2019-02-19 17:13:45.118 [20751] main/101/slave2 I> assigned id 1 to replica 825af7c3-f8df-4db0-8559-a866b8310077 2019-02-19 17:13:45.118 [20751] main/101/slave2 D> memtx_tuple_new(40) = 0x7f99a4000500 2019-02-19 17:13:45.118 [20751] main/101/slave2 I> assigned id 2 to replica 403c0323-5a9b-480d-9e71-5ba22d4ccf1b 2019-02-19 17:13:45.118 [20751] main/101/slave2 I> recover from `/var/lib/tarantool/slave2/00000000000000000014.xlog' 2019-02-19 17:13:45.118 [20751] main/101/slave2 I> done `/var/lib/tarantool/slave2/00000000000000000014.xlog' He even started:
2019-02-19 17:13:45.119 [20751] main/101/slave2 D> systemd: sending message 'STATUS=running' But for unknown reasons, he lost the connection and fell:
2019-02-19 17:13:45.129 [20751] main/101/slave2 D> SystemError at /build/tarantool-1.10.2.146/src/coio_task.c:416 2019-02-19 17:13:45.129 [20751] main/101/slave2 tarantoolctl:532 E> Start failed: /usr/local/share/lua/5.1/http/server.lua:1146: Can't create tcp_server: Input/output error Attempting to start the slave again does not help.
Now I delete the files created by the instances. In my case, I delete everything from the / var / lib / tarantool directory.
I start the slave first, and only then master. And lo and behold ...
# ps -ef | grep tara taranto+ 20922 1 0 17:20 ? 00:00:00 tarantool slave2.lua <running> taranto+ 20933 1 1 17:21 ? 00:00:00 tarantool master.lua <running> I did not find any explanation for this behavior, except as a "feature of this software."
This situation will appear every time if your server has completely rebooted.
Upon further analysis of the architecture of this software, it turns out that it is envisaged to use only one vCPU for one instance and many more resources remain free.
In ideology from n vCPU, we can raise the master and n-2 slaves for reading.
Considering that on a test server of 8 vCPU we can raise a master and 6 instances for reading.
I copy the file for the slave, correct the ports and start, i.e. added a few more slaves.
Important! When adding another instance, you must register it on the master.
But you must first start the new slave, and only then restart the wizard.
Example
I already had a configuration running with a master and two slaves.
I decided to add a third slave.
I registered it on the master and restarted the master first, and this is what I saw:
# ps -ef | grep tara taranto+ 20922 1 0 Feb19 ? 00:00:29 tarantool slave2.lua <running> taranto+ 20965 1 0 Feb19 ? 00:00:29 tarantool slave3.lua <running> taranto+ 21519 1 0 09:16 ? 00:00:00 tarantool master.lua <orphan> Those. our master became a loner, and replication collapsed.
Running a new slave will no longer help and will end in an error:
# systemctl restart tarantool@slave4 Job for tarantool@slave4.service failed because the control process exited with error code. See "systemctl status tarantool@slave4.service" and "journalctl -xe" for details. And in the logs I saw a record of little information:
2019-02-20 09:20:10.616 [21601] main/101/slave4 I> bootstrapping replica from 3c77eb9d-2fa1-4a27-885f-e72defa5cd96 at 10.78.221.74:3301 2019-02-20 09:20:10.617 [21601] main/106/applier/replicator@tarantuldb-t I> can't join/subscribe 2019-02-20 09:20:10.617 [21601] main/106/applier/replicator@tarantuldb-t xrow.c:896 E> ER_READONLY: Can't modify data because this instance is in read-only mode. 2019-02-20 09:20:10.617 [21601] main/106/applier/replicator@tarantuldb-t D> => STOPPED 2019-02-20 09:20:10.617 [21601] main/101/slave4 xrow.c:896 E> ER_READONLY: Can't modify data because this instance is in read-only mode. 2019-02-20 09:20:10.617 [21601] main/101/slave4 F> can't initialize storage: Can't modify data because this instance is in read-only mode. Stop the wizard and start a new slave. There will also be an error, as on the first launch, but we will see that it has loading status.
# ps -ef | grep tara taranto+ 20922 1 0 Feb19 ? 00:00:29 tarantool slave2.lua <running> taranto+ 20965 1 0 Feb19 ? 00:00:30 tarantool slave3.lua <running> taranto+ 21659 1 0 09:23 ? 00:00:00 tarantool slave4.lua <loading> But when you start the master, the new slave drops, and the master does not go with the status of “running”.
# ps -ef | grep tara taranto+ 20922 1 0 Feb19 ? 00:00:29 tarantool slave2.lua <running> taranto+ 20965 1 0 Feb19 ? 00:00:30 tarantool slave3.lua <running> taranto+ 21670 1 0 09:23 ? 00:00:00 tarantool master.lua <orphan> In this situation, the only way out. As I wrote earlier, I delete the files created by the instances and run slaves first, and then master.
# ps -ef | grep tarantool taranto+ 21892 1 0 09:30 ? 00:00:00 tarantool slave4.lua <running> taranto+ 21907 1 0 09:30 ? 00:00:00 tarantool slave3.lua <running> taranto+ 21922 1 0 09:30 ? 00:00:00 tarantool slave2.lua <running> taranto+ 21931 1 0 09:30 ? 00:00:00 tarantool master.lua <running> Everything started successfully.
That's how trial and error figured out how to configure and run replication correctly.
As a result, the following configuration was collected:
2 servers.
2 master. Hot reserve.
12 slaves. All active.
In the logic of tarantool, http.server was used so as not to fence an additional adapter (remember about the vendor, platform and soap) or to attach a library to each business process.
In order to avoid discrepancies between the masters, on the balancer (NetScaler, HAProxy or any other your favorite) we prescribe the rule of reserve, i.e. insert, update, delete operations go only to the first active master.
At this time, the second simply replicates the records from the first. The slaves themselves connect to the first specified master from the configuration, which is what we need in this situation.
On lua implemented CRUD operations for key-value. At the moment, this is enough to solve the problem.
In view of the peculiarities of working with soap, a proxying business process was implemented, in which the logic of working with a tarantula via http was laid.
If the key data is present, then it is returned immediately. If not, then the request goes to the master system, and is saved in the Tarantool database.
As a result, one business process in tests handles up to 4k requests. At the same time response from the tarantula ~ 1ms. Average response time up to 3ms.
Here is some information from the tests:
There were 50 business processes that go to 4 master systems and cache data in their memory. Duplication of information in full growth on each instance. Considering that java loves memory anyway ... perspective is not the best.
Now
50 business processes request information through the cache. Now information from the 4 instances of the wizard is stored in one place, and is not cached in memory on each instance. It was possible to significantly reduce the load on the master system, there are no duplicates of information and the memory consumption on the instances with business logic has decreased.
An example of the amount of information stored in the memory of a tarantula:
At the end of the day, these figures may double, but there is no “drawdown” in performance.
In combat, the current version creates 2k - 2.5k requests per second of the real load. Average response time is similar to tests up to 3ms.
If you look at htop on one of the servers with tarantool, then we will see that they “cool off”:
Results
Despite all the subtleties and nuances of the Tarantool database, you can achieve great performance.
I hope this project will be developed and these uncomfortable moments will be resolved.
Source: https://habr.com/ru/post/441686/
All Articles