RocksDB server - fast key-value storage for SSD drives

RocksDB is a permanent key-value storage for fast drives. Its main purpose is to store data on flash drives.

A performance bottleneck is often database access.
This problem can be solved in different ways.
Using the cache solves the performance problem, but significantly complicates the program architecture. Graph database out of the situation due to the optimal algorithms for this task. Another type of solution is storage, achieving high performance through the use of fast media.
Recently, a lot of NoSQL repositories have appeared that completely store data in memory. But the memory is still expensive and its volume is limited. The increase in memory due to sharding again rests on the cost.
The logical way out would be to use SSD drives. They have a relatively low cost and at the same time quite short response time.


')
Unfortunately, despite the fact that SSD drives have been around for several years, not many databases are optimized to work with them. The main problem when using an SSD disk as a carrier for a database is multiple rewriting of the same blocks. T.N.Z. "Freezing".

RocksDB is a flexible, productive NoSQL embedded storage. It is designed for use on fast media, such as SSD drives. It is written in Facebook and is based on LevelDB Google.
If you don’t go into details, the main differences from LevelDB are the engine optimized for working with flash drives; optimization for working with large amounts of data; Greater flexibility and expandability and, as a result, many tasty buns that are absent in LevelDB.

As I said, RocksDB is an embedded solution. And in order to be able to use it, you need to either build it into your application, or wrap it in a server. I needed it for use in web applications. And, for certain reasons, due to the fact that only one process (but not the thread) can have access to it at the same time, I just needed a server.
I learned about RocksDB about a year ago and waited patiently for someone to start writing a server for it. All that appeared during this time did not fully suit me. So I decided to write my own server wrapper over RocksDB.

RocksServer is a single- stream http server that provides access to RocksDB.

The server is written in C ++. Libevent was used to implement the http layer.
In addition to Libevent and RocksDB, the code does not contain any other dependencies.

RocksDB has very rich features. I have implemented only the basic functionality I need at the moment. In the future, of course, as far as possible, I plan to expand the functionality of the server.

I did not conduct large-scale benchmarks, but preliminary measurements of performance showed quite decent results.

Currently supported operations:

Get	get one value by key
Multi get	atomically get multiple values ​​on a set of keys
Set	set one value by key
Multi set	atomically set multiple values ​​on a set of keys
Delete key	remove key from DB
Multi delete	atomically delete multiple keys
Check key exist	quick check of the existence of a key in the database (with the ability to quickly retrieve values)
Imcrement	atomic increase / decrease values ​​by a given value

If necessary, the list of operations can be easily expanded. To do this, just need to implement the interface:

struct RequestBase { virtual ~RequestBase() {} /** * Runs request listener * @param event request object * @param event buffer object */ virtual void run(const EvRequest &, const EvBuffer &) = 0; }; 

For example, depending on your data schema, you may need to implement an atomic keyset insertion / update.
Connecting your handler to the server is also not difficult. This is done in one line:

 server.onRequest("/castom_handler", new Requestastom()); 

Compile / Install

As I said, RocksServer has two dependencies: RocksDB itself and http layer in the form of Libevent. You can install them separately, or you can do without installation by limiting compilation (due to static binding). For me personally, the latter method is preferable since it allows you to run RocksServer including. on old servers that do not have all the necessary dependencies.

So.
Clone the repository:

 git clone --recursive git@github.com:valmat/RocksServer.git cd RocksServer 

or

 git clone git@github.com:valmat/RocksServer.git cd RocksServer git submodule update 

After cloning, Libevent and RocksDB will be in the deps directory.
Then compile the dependencies:

 ./deps/make.sh 

After this step, you can, if you want, run the tests or install Libevent and RocksDB on your system.
And you can immediately go to the compilation of RocksServer

 cd src make 

Your compiler must support C++11 standard.
Immediately after compiling RocksServer, you can start it by editing the configuration file:

 ./RocksServer.bin config.ini 

And you can install it.

 make install 

In the latter case, you need to use the init.d script to start / restart / stop:

 /etc/init.d/rocksserver start 

Most likely, you will want to configure the server for yourself. All available settings are recorded in the config.ini file.

For testing, I put a simple driver in php ( yes, I know that it needs to be rewritten ), or you can directly use the protocol . Including using programs such as Curl and Wget .
The protocol is very simple. When choosing between RPC and HTTP, the simplicity of the exchange protocol was one of the decisive moments.

At the moment, despite the not very rich functionality, RocksServer is completely ready for work. In any case, I plan to use it in production in the project I am working on now (and wrote for this).

Except for me, Valgrind and CppCheck, nobody conducted a review of the code. Therefore, the fear that something might go wrong is quite normal.
In this case, there is the possibility of logging RocksServer and the ability to make backups, including in human-readable format.

To enable logging in the configuration file, you just need to specify the log file:

 ; Error log file name ; default value: /var/log/rocksserver/error.log ; ; error_log = 

And indicate the level of log details:

 ; Error level ; Possible values: debug | msg | warn | error | fatal | none ; default value: none ; ; log_level = 

For backup, you need to specify the directory in which the backups will be added:

 ; RocksDB backup path ; The directory in which a backup will be stored ; default value: /var/rocksserver/backup ; ; backup_path = 

and perform a POST request to 127.0.0.1:5577/backup

To restore the database from the backup, use the command

 restore -f/path/to/backup -t/path/to/db 

To convert a database to a human readable form:

 human_readable -f/path/to/db -t/path/to/restore_file 

I am planning the following backup scheme:

 wget "http://localhost:5577/backup" --post-data="" -O - restore -f/path/to/backup -t/path/to/db_on_hdd human_readable -f/path/to/db_on_hdd -t/path/to/restore_file 

I would appreciate constructive criticism.
Ready to help with installation and configuration.
Well, of course, Pull Requests are welcome.
Thanks for attention.