Tarantool as the main data storage for server applications written in .NET

Hi, Habr! Today I want to share with you the text version of the report presented at the Tarantool Meetup on March 2, 2017 in the Mail.Ru Group with an amendment to what has already passed a month, and some of the promises have already been implemented, so the text will be interesting even for those who saw the performance. I work for eVote, which develops an online survey service. We actively use Windows and .NET-technologies in our products, and in this post I will tell about how we added Tarantool to the stack of our technologies.

Many thanks to colleagues, friends and employees of Mail.Ru Group who helped write this article.

Select DBMS

In the life of each project, sooner or later there is a turning point when you need to select a database for storing all data. From this point of view, our project is simple: users, polls, answers, some kind of incidentally collected information - all this can be perfectly stored in the key-value store. Therefore, at the start we considered three options: Redis, Tarantool and MySQL with handlersocket. The favorite from the start was Redis. It works fast, it has a great .NET connector, created by the Stack Overflow team. By the way, Stack Overflow itself is written in .NET, it works on Windows, they have SQL Server from Microsoft, Redis and many more interesting things . Redis has excellent documentation . If we hire a new programmer who has never worked with Redis, then we send him there - and in three days he knows about everything he needs to know in order to use Redis.

Under the second number was Tarantool. Unfortunately, he did not have such a convenient site as Redis. Like the connector for .NET. In terms of speed, it completely satisfied us, since it was not very different from Redis. If you enable write ahead logging, then any entry will be on the disk. And if your controller, disk or other hardware is not buggy, it turns out quite reliably. Also in Tarantool there are secondary indices. In some cases, this is a very important feature. It’s not in Redis, you have to do it manually.

Handlersocket turned out to be an outsider. It is slower than Redis and Tarantool. But the ACID model is available if your engine in MySQL supports it. All infrastructure from MySQL is available: replication, monitoring, eksplen, backups. You can build complex reports in plain SQL. There was no connector under .NET either.

As a result, we chose Redis and launched it into our closed alpha. But after some time it turned out that the absence of secondary indexes is a more serious problem than we thought. If you need to select all the polls created by any author, then you have to create additional lists for storage (you can, of course, do a full brute force, but this is not our case). There is a risk that as a result of various failures, the data will become inconsistent. Attempts to fix it through Lua scripts or transactions led to the fact that the work of Redis slowed down about three to four times, which ceased to suit us. This is not a problem of Redis itself, but a consequence of its inappropriate use, in my opinion.

Tarantool and Windows

Then we decided to go to Tarantool. And we faced two problems at once. Tarantool still does not have a binary under Windows (and it is not known when it will appear) - once. There was no connector for .NET - two. Generally speaking, it is a controversial question, can we call a lack of a version of our storage under Windows a disadvantage? I think this is an advantage. After all, production will most likely be Linux, and the lack of a Windows version will force the programmer to understand how everything actually works. Programmer for debugging and monitoring will use the same tools as in production. According to my observations, this increases the likelihood of writing high-quality code without errors and reduces downtime in the event of disasters.

The first problem we solved was using Docker for Windows for development. But the second problem was more complicated.

Connector

Since there was no ready-made connector for Tarantool, we wrote our own. For this we had to solve two problems. First: implement serialization and deserialization in msgpack , since this is the data exchange format in Tarantool. We have solved it in the framework of the MsgPack.Light project, since we also store data packaged in msgpack. The second task was reduced to the literal implementation of the data exchange protocol with Tarantool iProto . It is solved in the framework of the project progaudi.tarantool .

We support

• .NET 4.6 and up,
• New opensource framework. NET Core: netstandard 1.4 and above.

Previously, the connector was called tarantool.csharp, but after feedback from the community, we renamed it progaudi.tarantool. Now the name should not cause any confusion - whether it is possible to use a connector from F # or not. It was possible from the very beginning.

Thanks to recent improvements in MsgPack.Light, the connector has learned to work with the tarantula data type scalar. In the future, it is planned to further simplify the work with the connector and get away from explicitly converting user objects into TarantoolTuple-structures.

We do not support DDL, because it is beyond the scope of iProto and must be implemented by wrappers over the EVAL command. Personally, I think that if a programmer uses Tarantool, then he should write a scheme in Lua, because in this case, you can fumble with admins and distribute it directly into the working environment. Maybe I'm wrong, and we will implement the ability to do this from .NET.

At the moment we have a connection with only one Tarantool node. We do not fully support CALL_16, in certain cases strange packaging occurs on the Tarantool side, and this leads to errors during deserialization. The new CALL is fully supported, we recommend using it.

Features of the development of connectors to Tarantool

Unfortunately, in Tarantool it is impossible to ask the server to output all requests to the log. In some situations, this greatly interferes with debugging, and the performance of the Tarantool server is usually not important to us. I would like to get some kind of pen that includes the recording of all requests in the log.

When logging requests to the log, I want to somehow separate them from each other. For this we have a connection id (box.session.id) and a request id (box.session.sync). Unfortunately, box.session.sync is shared for all requests within a single connection, as a result, it can change if the execution of the request is interrupted due to the achievement of a yield point ( writing to the base, manual control and so on ). In principle, this is important only where several logging points may occur, for example, in stored procedures. In such cases, the box.session.sync should be saved before the first yield point in a local variable.

Apps server

An important part of Tarantool is the application server. There are many ready-made modules for it, starting with simple ones, such as automatic reloading of other modules , ending with sharding , queues, and drivers for other DBMS ( 1 , 2, and more). All the modules that we have tried, work fine, well documented and supported.

Of all this diversity, we use tarantool / queue for queues and tarantool / prometheus to collect metrics. We also have a little bit of our own logic on Lua. A little - because our team from the world of .NET. We are used to static code analysis, step-by-step debugging, convenient profilers. Lua has problems with this, especially with static code analysis.

Replication

I'd love to get synchronous replication. We already know how to live without her, but we are waiting very much. So far, we are using the existing master-master asynchronous replication. One of our services works with pictures. Random keys appear there, and we can write to any node, because the probability of coincidence of keys is extremely small. We need to match the generated guid for the images and SHA256. Since we didn’t come to this right away, the connector still does not know how to connect to several nodes. This year we will definitely fix it.

Cluster build in old versions

In a very old build 1.7.3-0, the cluster may not get together. Suppose you have three master nodes. Give them the same configuration, run. And until any of the nodes sees its sources, from where it takes the data, it will not accept requests from other clients. Unfortunately, these sources for them - customers. As a result, all three nodes do not respond to requests and wait 30 seconds. At this time, they are looking for their sources, they don’t find, they write to the log: “There are no sources, I turned off.” And the cluster is not going to. It is necessary to collect by hand. We start one note without the general configuration. It rises, we connect other nodes to it, and then we change the configuration to a single one. In the new version this bug is already fixed.

Docker versioning tarantool / tarantool

Suppose the version of the image is 1.7.3. And what version of Tarantool inside? We know that 1.7.3, but the build number is unknown. The only option is to view the source. Suppose you need a version of Tarantool 1.7.3-115, because at 114 there was still a problem that hurts us, and at 116 we don’t know if it is there or not. What version of the image to take?

We solved the problem simply: we assemble our image ourselves, indicate the specific assembly number - and everything works fine. But overall this is a minor issue. The default image works fine and will cover most of the requests, including all the main modules that are needed to work with Tarantool: monitoring, queues, etc. You can take and use it.

Interactive requests

To write an interactive query in Tarantool, you need to know Lua and be able to use the command line. Our testers and admins know how to handle the command line, but they are not eager to deal with Lua. They want to quickly write a SQL script, check some counter or timestamp. Therefore, we are looking forward to the announced support for the SQLite dialect.

Monitoring and logging

Everything is fine with them. There is a tarantool / prometheus module , which is published on the Prometheus website. As we use in production docker, we collect logs from it using Fluentd ( 1 and 2 ) and add them to ElasticSearch .

findings

.NET is not only Windows: .NET Core works on all platforms. We have a .NET application in production that functions perfectly on Linux. Our opensource-projects without problems gather and work on Windows, Mac OS and Linux (only Windows 10, Max OS X, Ubuntu 16.04 are tested). It is quite possible to use all the rich tools available in the .NET world for development and debugging, and often free, even for commercial development .

Thanks to the cross-platform new .NET solutions, it is possible to use previously inaccessible tools, for example, Tarantool. If you care about speed, secondary indexes, the ability to use a DBMS as an application server or task queue, then Tarantool is a very good choice today.