YAWNDB - time series database

As part of our many services, we need to regularly provide clients with various statistical information. Clients renting dedicated servers require information on traffic consumption. Cloud server users need statistics on the use of hardware and network resources, and cloud storage users need statistics on file downloads.

The simplest and most visual way of presenting statistical data is graphing. There are many specialized software solutions for analyzing statistical data with subsequent visualization. We began to look for a suitable tool; The main requirement was high performance. As a result of these searches ... However, everything is in order. Let's start with a little theoretical introduction.
')

Some theory

On any graph of network activity, changes in some parameters are displayed over a certain period of time (for a month, for a week, for a day, etc.). To build a graph, it is necessary to process statistical material that is a combination of time-value pairs for a specified time period. the gap. Such material is called a time series (English time series).

There are quite a few software tools for analyzing time series. Their common feature is the use of circular databases (English Round Robin database). A ring is a database in which the amount of stored data does not change with time, because the amount of data is constant: the database cells are cycled.

One or several archived data sets are stored in the ring database (English Round Robin Archives, RRA). In their structure, ring tables are similar to arrays whose last element has the same address as the first element. The position of the last updated item is stored as a pointer. Archives are interconnected so that each subsequent archive stores information from the previous one: one archive saves data with a small interval between records, the other after a specified number of intervals saves consolidated data from the previous one, the third makes it even less often and so on.

To do this, use the consolidation functions that are built into the database and are automatically applied when the information is updated. Consolidation functions are defined as obtaining the minimum, maximum, average, and total values ​​for a specified period of time. The consolidation of data in the ring bases is carried out during the recording and not during the reading (this ensures a high speed of work).

RRDTool

The most famous and common tool for analyzing time series and subsequent visualization is, of course, RRDTool.

We tried to use RRDTool in our own practice, however, for a variety of reasons, it did not suit us - primarily because it did very poorly with the loads.

When the number of files to which we recorded data exceeded 1000, problems began to arise: for example, the recording of these same data began to take too long. Sometimes the data was simply not recorded, although no errors or interruptions occurred.

These facts are quite enough to conclude that RRDTool does not suit us at all: with the number of virtual machines we have in the cloud, tens of thousands of write operations per second are required.

You can reduce the number of write operations to disk when working with large amounts of data in the RRD Tool using the RRDcacheD daemon, which caches data and, after accumulating a certain amount, writes to the database. However, practice has shown that RRDcacheD is poorly suited for solving our problems.

Accumulating the data, it at the same time does not allow to read them from the cache, but only writes to the disk. If you need to somehow process the data, then you need to write everything to disk and then read from disk. The larger the amount of data, the worse the cache works: the hard disk is heavily loaded, an additional load on the processor is created ...

Another feature of RRDcacheD is that it writes data to disk at the most unexpected and inappropriate moment.

An argument against RRDTool was the inability to change the ring database settings. Of course, this can be done by exporting data, creating a file with new parameters and then importing old data into it, but this method is too inconvenient and time consuming.

Faced with all the difficulties described, we decided not to use RRDTool. We also tried other means of processing and visualizing data - for example, graphite, which did not suit us due to poor performance.

In the course of exploring the existing tools, we all understood more clearly that none of them fit our specifics. We had a desire to develop our own solution that fully meets all our requirements. The main requirements for the product were, firstly, flexibility, customizability and the ability to adapt to the specifics of our services, and secondly, high performance. So there was a ring in-memory database YAWNDB.

YAWNDB: general information

YAWNDB (this name with the English verb to yawn - “yawn” - has nothing in common; the acronym YAWN means Yet Another iNvented Wheel) is an in-memory database; All data is stored in memory and periodically written to disk. It is written in Erlang. The lightweight process model underlying this language allows for fast processing of large data sets with a small consumption of system resources.

The data arriving in YAWNDB is divided into archives (in the terminology of YAWNDB, also called buckets) in accordance with some rules.

A rule is a set of properties for a particular statistic (data size, collection period, etc.).

All these data are triplets, consisting of time, value and key (the key in YAWNDB terminology is also called path). The path is a sequence of lowercase letters and numbers that determines exactly where the triple should be saved. In this sequence, the first component is the most important - the prefix. The rule also includes the Prefix field and thus determines how data is stored for a specific prefix. For the same prefix, you can create multiple rules. In this case, the triplet will be recorded in the database in accordance with each of these rules.

This means that a specific time-value pair will go to N baskets, where N is the number of rules corresponding to the path prefix for the pair.

The approach we use provides the following benefits:

• deletion of data that has become out of date without wasting resources;
• fixed memory consumption for a certain number of keys;
• fixed access time to random write.

Internal organization

YAWNDB is based on the round-robin algorithm implemented in the C language of Ecirca.

Other modules written in Erlang interact with it using NIF (Native Implemented Functions). Bitcask Erlang application is used to save data to disk.

The REST API is based on the Cowboy web server.
Data is written through a socket, and read through a REST API interface.

Principle of operation

For each path, a separate process path N is created . This process stores data directly, as well as various service information.
Each incoming request for receiving data is processed by the free request handler N (if there is no free handler at the time of the request, a new one is created). The resulting parameters are passed to the path manager, who retrieves the required data from the corresponding processes.

The process of dumper is responsible for saving to disk. It conducts a periodic survey of all the processes of the paths and saves the data that has been changed since the last save.

Installation

To work with YAWNDB, you must install the LibYAML parser. Then you need to clone the repository:

$ git clone git@github.com:selectel/yawndb.git 

And execute the following commands:

 $ cd yawndb $ make all 

The launch of YAWNDB is performed using the command:

 $ ./start.sh 

Before launching, you need to copy the sample configuration file to the place of this:

 $ cp priv/yawndb.yml.example priv/yawndb.yml 

Or create an appropriate symlink.

Configuration

All YAWNDB settings are stored in the yawndb.yml configuration file.
As an example, consider the configuration file that is used to process statistics in our cloud storage:

 rules:
     # User statistics
     # detailed statistics by the minute (for 24 hours)
     - name: per_min
       prefix: clientstats
       type: sum
       timeframe: 60
       limit: 1440
       split backward
       value_size, large
       additional_values: []

     # stats by hour (for the last month)
     - name: per_hour
       prefix: clientstats
       type: sum
       timeframe: 3600
       limit: 720
       split: backward
       value_size: large
       additional_values: []

     # statistics by day (stored for the last two years)
     - name: per_day
       prefix: clientstats
       type: sum
       timeframe: 86400
       limit: 730
       split: backward
       value_size: large
       additional_values: []

Examples of using

Build a package to write data (Python)

 def encode_yawndb_packet(is_special, path, time, value): """       yawndb. :param bool is_special:  ?   additional_values :param str path:   :param int value:   """ is_special_int = 1 if is_special else 0 pck_tail = struct.pack( ">BBQQ", YAWNDB_PROTOCOL_VERSION, is_special_int, time, value ) + path pck_head = struct.pack(">H", len(pck_tail)) return pck_head + pck_tail 

Formation of a packet for data recording on

 //    #define YAWNDB_PROTOCOL_VERSION 3 //    struct yawndb_packet_struct { uint16_t length; uint8_t version; int8_t isSpecial; uint64_t timestamp; uint64_t value; char path[]; }; //   yawndb_packet_struct *encode_yawndb_packet(int8_t isSpecial, uint64_t timestamp, uint64_t value, const char * path) { yawndb_packet_struct *packet; uint16_t length; lenght = sizeof(uint8_t) + sizeof(int8_t) + sizeof(uint64_t) + sizeof(uint64_t) + strlen(path); packet = malloc(length + sizeof(uint16_t)); packet->length = htobe16(length); packet->version = YAWNDB_PROTOCOL_VERSION; packet->isSpecial = isSpecial; packet->timestamp = htobe64(timestamp); packet->value = htobe64(value); strncpy(packet->path, path, strlen(path)); return packet; } 

Conclusion

Today, YAWNDB is used by us both in public services and in internal projects. The source code of the project is available on GitHub. We will be happy if one of our readers takes advantage of our product. We will also be very grateful for comments and suggestions for improving it.

On March 28 we published a site dedicated to our open source development: selectel.io . On it you can find detailed documentation for YAWNDB in Russian , as well as information about our other projects.

Readers who for one reason or another cannot comment on posts on Habré are invited to our blog .