We in Mail.Ru Mail are constantly faced with the need to work with the history of users. Given that the monthly audience of the project is more than 40 million people, the history of all their actions is in the order of petabyte data. The need to search for logs arises hundreds of times a day, and on average it took several hours to obtain the necessary information. In this case, according to our assumptions, the extraction of information from the logs could be accelerated to a few seconds.

To evaluate the feasibility of developing a system for optimizing the search by logs, we used this table with XKCD :


')
(not really, but we still like it).

So, we seriously took up the optimization. The result of our work was the development of a system, thanks to which we can raise the history of actions by about 100,000 (one hundred thousand, this is not a typo) times faster. We have developed a big-data service that allows you to store petabytes of information in a structured way: we have a log of some events for each key. The storage is designed so that it can work on the cheapest SATA disks, and on large multi-disk storages with a minimum of processor time, while it is completely fault-tolerant - if suddenly a machine fails, it doesn’t matter affects. If the system runs out of space, it simply adds a server or several: the system automatically sees them and starts recording data. Reading data occurs almost instantly.

What's the catch?

Why do you need it? Suppose the user is not working. To reproduce the error, we need to know the whole background: where the person went, what buttons and links clicked. Also, logs are necessary for decision-making in cases such as hacking a user account, when it is important to separate the actions of the user and the attacker.

Accordingly, the speed of the response of the technical support service is “tied” to the speed of the search by logs.

During a working day, a large system can accumulate several terabytes of logs, especially considering that our company has a lot of “compound” services. Suppose the user went to the mail, the mail turned to the database, the database to the storage system, she also wrote logs ... To understand what the user was doing, all this data needs to be collected into a heap. To do this, use the good old grep command, from which the word "grepat". To solve the problem of a single user, we for example, terabyte. This process can take several hours. We attended to this problem and began to think how to process this data much faster.

The problem is that there are no complete separate systems for storing and processing logs corresponding to our tasks. For example, Hadoop (like any other implementation of MapReduce) can parallel logs in parallel and bring together the results. But, since in the Mail and Mail.Ru Cloud the volume of logs that need to be stored is estimated to be petabyte, even parallel processing of such a volume will require a lot of time. In addition, for a day, it is necessary to go through several terabytes, and for a week, this volume can increase to ten terabytes. Plus, if you need to bail out for a large number of different logs, then to solve just one problem you might need a few dozen terabytes of logs.

Hard disks can write information linearly at a speed of about 100 Mbps. However, quick writing implies unstructured, non-optimal placement of information. And for quick reading of files it is necessary that they be structured in a certain way. We needed to provide high speed of both processes. The logs constantly preserve a large amount of information, while we need to quickly collect statistics on a specific user in order to promptly solve his problems, providing high-quality technical support.

It would be possible to follow the simplest path by organizing the storage of logs in a database. But this is a very, very expensive solution, since it requires the use of SSD disks, sharding the database, creating replicas. We wanted to find a solution that would allow us to work with the most cheap storage on slow disks of huge volume.

Taking into account all these factors, it was decided to write a system from scratch, which allows speeding up the process of fastening. By the way, as far as we know, it has no analogues not only in Russia, but also in the world.

How did we do it?

The system we have created for this is as follows. From the programmer’s point of view, it performs essentially two operations: creating a record for a specific user (ID and time) and getting the entire history for a specific user.

The recording is carried out in our database Tarantool. In it all information is always stored in one file, and reading is carried out from memory. Base shardirovana on several instances. The logs themselves are stored on multidisk servers. Since Tarantool stores everything both in memory and on disk, it is clear that this cannot continue indefinitely. Therefore, a process is periodically launched that groups data from Tarantool by users and writes them into files on servers, and in the database puts a link to this data.

Thus, Tarantool stores only a recent history by users (accordingly, it can be quickly retrieved). And the “old” data is stored on disks in one large file, well structured and suitable for quick reading. Thanks to this, the entire history of any user can be obtained in a matter of seconds. For comparison: prior to this, weekly period for one user took about three days.

Each disk contains several files in which all information on different users is recorded in a row. So that Tarantool does not overflow, the following is done: a process that consistently reads all these files, takes the information about the user from the file, adds to it what is in Tarantool, and writes it all back to the end of the file. This whole system is also shardirovanny and replicated, that is, there can be as many servers as possible, because every time we rewrite the user's history, we always write it down to two disks, which are chosen arbitrarily. After successful pair recording, the link is saved in Tarantool. Thus, if even one of the servers falls, the information does not disappear. When a new server is added, it is registered in the system configuration, after which new data is automatically saved to it.

Now this system is used in Mail and Cloud Mail.Ru. In the near future, we plan to implement it in Mail.Ru for Business, and then gradually on other projects.

How it works?

We search the logs in two types of cases. First, to solve technical problems. For example, a user complains that something is not working for him, and we need to quickly see the whole history of his actions: what he did, where he went, and so on. Secondly, if we have a suspicion that a box has been hacked. As I said above, in this case, the story is needed in order to divide the user's actions into his own and those committed by the attacker.

I will give a few examples of how effective the system we have created.

Quite a frequent case: the user complains about the loss of letters from the mailbox. We pick up the story and see when and from which programs he deleted them. Suppose we see that the letters were deleted from such an IP address via POP3, from which we conclude that the user connected the POP3 client and selected the option “Download all letters and delete them from the server” - that is, in fact letters were deleted by the user, and not "disappeared." Thanks to our systematized system of log storage, we can establish this fact and demonstrate it to the user in a matter of seconds. Saving time admins and support - a giant.

Another example: we receive a complaint from a user who cannot open the letter. Having raised all the logs for this user, we can literally within a few seconds see when and which server calls were made, by what user agents, what was happening on the client side. Thanks to this, we can more easily and, most importantly, establish more quickly what the problem is.

Another case - the user complains that he can not enter the mailbox. We begin to look for the cause - it turns out that the account was automatically blocked for spamming. Here you need to figure out who exactly is spam - the user himself or someone else on his behalf, and for this you will have to analyze the entire history of using the box for a long period (perhaps a week, a month or even more). Having glanced in history, we see that, starting from a certain date, they came into his box from a suspicious (ie, “non-native” for the user) region. Obviously, he was just hacked. Moreover, the data collection and analysis took not a day, but several seconds: the technical support employee simply pressed a button and received exhaustive information. The user changed the password from the mailbox and got access to it again.

In short, our system saves us a lot of time every day, while allowing us to reduce the response time to the user - in my opinion, this is win. We will tell about other ways of its application in the following posts.