Tarantool: Good, Bad, Angry

Many have heard of the Tarantool NoSQL database, they know that it can store data in memory, processes it very quickly and has high performance. Tarantula was written by serious guys who serve services with hundreds of thousands of requests per second.



The system seems complicated. Despite Russian roots, initially there was not even documentation in Russian. What can help this powerful tool for ordinary guys - programmers and novice developers? The rest can immediately see the result.



Let's try to write a simple entertaining service that can withstand a large load. And no SQL!

Our goal

To fuel interest in the learning process, take a simple and interesting example of using Tarantula in web development. We will launch the site, which displays a pair-wise vote for some pictures. We will not upload photos of college students, like Mark Zuckerberg, but we will arrange a vote on the choice of a sticker for the Telegram messenger. Our algorithm by voting will choose the top of the best stickers, but in reality our task is to find the main monster of the collection - Mr. The Ugly. Now that there is motivation, it's time to do the boring things.

')

Installation

To crow somewhere god sent a piece of cheese. We also got a modest virtual server with one processor core and 1 GB RAM - the younger offspring from a well-bred German family. The operating system is Debian, so we can install Tarantula from the official repository: tarantool.org/download.html



The manual says Copy and Paste, which we do with great enthusiasm:

curl http://download.tarantool.org/tarantool/1.7/gpgkey | sudo apt-key add - release=`lsb_release -c -s` # install https download transport for APT sudo apt-get -y install apt-transport-https # append two lines to a list of source repositories sudo rm -f /etc/apt/sources.list.d/*tarantool*.list sudo tee /etc/apt/sources.list.d/tarantool_1_7.list <<- EOF deb http://download.tarantool.org/tarantool/1.7/debian/ $release main deb-src http://download.tarantool.org/tarantool/1.7/debian/ $release main EOF # install sudo apt-get update sudo apt-get -y install tarantool 

A little surprise from the developers: the i386 architecture is not supported in the official repository, although the package hosting service regularly provides all the files except the main one. Make sure you install the packages on the AMD64 system to avoid trouble.

Beginning of work

After installation, we have a process tarantool , launched with a test configuration example.lua, like this:

 ps xauf | grep taran root 2735 0.0 0.2 13972 2132 pts/0 S+ 22:05 0:00 \_ grep taran taranto+ 568 0.0 0.8 812304 8632 ? Ssl 17:03 0:03 tarantool example.lua <running> 

To work with Tarantool, the Lua scripting language is used, in which server control commands are written, stored procedures and triggers are described. You can load ready-made software modules or write your own.



Run the tarantoolctl utility, print the first program and make sure that everything works.

 tarantoolctl connect '3301' connected to localhost:3301 localhost:3301> print ('The good!') --- ... localhost:3301> 

Now we are in the test configuration that comes with the distribution. According to the Debian ideology, the settings are in the /etc/tarantool/instances.available/* directory, and the program's configuration is launched using a symbolic link in the /etc/tarantool/instances.enabled/* directory. Copy the sample file under the new name and create our project.



Our project is called the good, the bad and the ugly , in abbreviated form - gbu . Use the abbreviation of the full name of the project to the first letters, and your colleagues will always treat your work with respect and awe!



Now fix gbu.cfg a little and start the service. Let me remind you that the configuration uses the syntax of the Lua language, in which comments begin with two hyphens.

 box.cfg { --          listen = 3311; --       slab_alloc_arena = 0.2; --      } local function bootstrap() local space = box.schema.create_space('example') space:create_index('primary') --     -- box.schema.user.grant('guest', 'read,write,execute', 'universe') --    box.schema.user.create('good', { password = 'secret' }) box.schema.user.grant('good', 'read,write,execute', 'universe') end --         box.once('example-2.0', bootstrap) 

We start the new instance with the tarantoolctl start gbu command and make sure that everything works as it should:

 tarantoolctl connect "good:secret@0:3311" connected to 0:3311 0:3311> 

We are in business!

Database

• Records in Tarantool are stored in spaces (space), this is such an analogue of a table in a relational SQL database. They can be done as long as necessary - up to 65 thousand
• Inside the space are tuples, which are similar to both the row in the SQL table and the JSON array of data. The maximum size of the tuple is up to 1 MB with the default settings. This parameter can be changed.
• In order for the database to be useful, it is necessary to create indexes, just like in SQL. Thanks to them, instead of going through the complete list of all the elements, the search is performed using a faster algorithm. Sorting is also available. The choice of index depends on the type of data.

HASH index : the value must be unique, and can be arbitrary. This is how well-known Key / Value-storages are organized, also known as Map. A good example is the MD5 hash checksum.



TREE-index : the value may be non-unique, but must be “dense” to organize a sorted list. It turns out an array (Array), which may have missing elements. A good example is the order number, which is incremented by one.



If you need a unique value, then you can use HASH or TREE, while HASH will be faster on sparse data. If you need a non-unique field by which sorting will be done, then only the TREE index can be used.



There are also RTREE indices for searching on a two-dimensional plane and BITSET for working with bit data, but we will not need them. Read more about this in the documentation .



Picture from Eugene Shadrin's article Mastering the Tarantula . By the way, a good guide for the initial dating.

Project Data Model

In the data model of our application, we will create a space stikers to store information about files. Please note that the numbering of the fields starts from 1, since the syntax of the Lua language is used. The tuple includes the following fields:

1. unsigned id - unique sticker number
2. integer rating - sticker rating
3. string pack - the name of the sticker pack
4. string name - the name of the sticker file
5. (string) path - URL of the sticker
6. (number) up - the number of votes per sticker
7. (number) down - the number of votes against the sticker.

In the packs space, we will store a list of sticker packs:

1. string pack - the name of the sticker pack
2. integer rating - sticker pack rating
3. (string) path - link to the description page

In the secrets space we will store a token to encrypt the link to the image in order to implement the simplest anti-cheat protection:

1. string token - random token for a sticker
2. integer time - token creation time (useful for deleting old ones)
3. (integer) id - unique sticker number (space key stickers)
4. (string) url - URL of the sticker

In the sessions space we will record visitors and collect statistics:

1. string uuid - unique character identifier of the visitor
2. integer uuid_time - session creation time (useful for deleting old ones)
3. (number) user_votes - how many times the visitor voted
4. (string) ip - IP address of the visitor
5. (string) agent - the type of the visitor's browser

In server space, we will simply collect site statistics:

1. Integer id - just key
2. (number) visitors - the number of unique users
3. (number) votes - the total number of votes
4. (number) clicks - the total number of clicks in the vote

Note that in order to assign an index, you must explicitly specify the field type. This type should be selected from the list of possible options for the Tarantula.



The remaining fields can be of any type that the built-in Lua interpreter supports. This dualism of data types is a feature of the Tarantula and is mentioned in the documentation . For convenience, we specified the Lua data type in parentheses when describing the model.



An important part in modeling is the compilation of indices. Tarantula’s big advantage is that it can make complex composite indexes. Due to this, we can write fast analytical queries based on various fields of the tuple without reducing the system performance. Add a primary index of type TREE for the id field to provide a random selection of the item for voting. The second index is of type TREE across the Raiting field, in order to display a rating, of course! Add a composite index on the fields Pack + Emoj type HASH, which must be unique. It can be used to analyze the popularity of sets of stickers.



We will place the database creation code in our gbu.lua file, in the initialization procedure





Before restarting the server with the settings, try creating a scheme with commands in the console. If something goes wrong, you can remove the entire space with the command: box.space.stickers:drop()

or a separate index: box.space.stickers.index.ternary:drop()



Feel free to use the TAB key prompt. For convenience of working in the console, we write the names of the created elements of the scheme with a small letter. Commands for working in the console become intuitive after a brief introduction to the documentation.

Clear space: box.space.stickers:truncate()

Delete space: box.space.stickers:drop()



Everything happens instantly, as it should be for the In-memory Database!

Component installation

A good modern programming language should have static strong typing, Homoiconicity is a property that allows you to manipulate code as data, support for the PLO, FFI to C libraries, support for generics, competitive programming, Functions as first-class citizens, lambdas.



None of this in PHP, of course, no! Therefore, we will write the code of the example on it.



First, we put the proven tools - Nginx web server and PHP interpreter - php-fpm : wiki.debian.org/ru/nginx/nginx+php-fpm



Add the query rewrite rule to the root configuration path of nginx:

 location / { try_files $uri $uri/ /index.php?q=$uri&$args; } 

Thus, we can get beautiful links of the form / good from the $_REQUEST['q'] array in the PHP script, and implement the routing of HTTP requests.



We also have a locale to execute CGI requests:

 location ~* \.php$ { try_files $uri =404; fastcgi_pass unix:/var/run/php5-fpm.sock; fastcgi_index index.php; include fastcgi_params; expires -1; } 

The command expires -1; we disable the caching of requests, it is not needed for the voting pages and the output of the Top Charts. The rest of the locale cache data for 24 hours or 30 days from the upstream HTTP settings. Probably, everyone has their own collection of Nginx options.



Now you need to put the module to work with the Tarantula:

 sudo apt-get install php5-cli php5-dev php-pear pecl channel-discover tarantool.imtqy.com/tarantool-php/pecl pecl install Tarantool-PHP/Tarantool-beta 

We read what is written in the output of the installer. In my case there was a message:

 Build process completed successfully Installing '/usr/lib/php5/20131226/tarantool.so' install ok: channel://tarantool.imtqy.com/tarantool-php/pecl/Tarantool-0.0.13 configuration option "php_ini" is not set to php.ini location You should add "extension=tarantool.so" to php.ini 

Add the specified line to the configuration file in the /etc/php5/fpm/php.ini file and /etc/php5/cli/php.ini . Unfortunately, when you start PHP, we get an error! In order not to suffer with debugging of the web server, we added a new library to the cli configuration, so you can check the functionality from the command line.

 php -v PHP Warning: PHP Startup: Unable to load dynamic library '/usr/lib/php5/20131226/tarantool.so' - /usr/lib/php5/20131226/tarantool.so: undefined symbol: tarantool_schema_destroy in Unknown on line 0 PHP 5.6.29-0+deb8u1 (cli) (built: Dec 13 2016 16:02:08) 

At the time of this writing, the module in the PEAR repository contained an error, so the only thing left is the Jedi way — building the driver from the source codes.

 pecl uninstall Tarantool-PHP/Tarantool-beta cd ~ git clone https://github.com/tarantool/tarantool-php.git cd tarantool-php phpize ./configure make make install 

Data loading

Let's create the first file and call it test.php , in it we will check the operation of our database.

 <?php $tarantool = new Tarantool('localhost', 3311, 'good', 'secret'); try { $tarantool->ping(); } catch (Exception $e) { echo "Exception: ", $e->getMessage(), "\n"; } ?> 

Run the php config.php from the command line and check how it worked. If configured incorrectly, we get an error message. Check out!



Now you can write a parser that will collect data from the site we need. We will explore tlgrm.ru/stickers . First, load the table pack , in which we have a census of sticker packs. This is what the insert command looks like on the tarantool command line:

 box.space.packs:upsert({'key1',0}, {{'=',2,0}}) 

This command adds the new key “key1” (in field 1) and the value 0 (in field 2). If a record exists, it is updated for the same record (= sign) in field 2 with value 0. As we remember, in field 2 we have a rating, which we initially use at 0. The upsert command is convenient to use for running the parser multiple times during debugging to do not delete the entered data each time. The PHP version of the command will look like this:

 $tarantool->upsert('packs', array ($pack,0), array ( array( "field" => 1, "op" => "=", "arg" => 0 ) )); 

Ah-ah! In PHP, the numbering of fields with 0, and in Lua c 1. Therefore, "field" => 1 from the PHP array corresponds to the {'=',2,0} entry in Lua. Wherever the arrays start from zero, the current connectors work the same way. This behavior has been changed since version 1.6. Reading examples on the Internet, pay attention to the version of the Tarantula! This article is written according to version 1.7, and about version 1.5, developers are asked not to remember at all.



We make the record for the sticker using the built-in auto_increment procedure, which automatically increases the primary index. Tarantula team:

 box.space.stickers:auto_increment({0,'pack2','sticker2'}) 

PHP:

 $tarantool->call('box.space.stickers:auto_increment', array( array(0,$pack, $i . '.png', $url, 0, 0) )); 

So, the script is written. Run it - vzhuh, and magic! Now we have a database with 16,000 entries!

We write the program

To begin with we will make the simplest query router, as is usually done in PHP:

 # Get routes from request $route = isset($_REQUEST['q']) ? $_REQUEST['q'] : '/'; $vote_plus = isset($_REQUEST['vote_plus']) ? $_REQUEST['vote_plus'] : ''; $vote_minus = isset($_REQUEST['vote_minus']) ? $_REQUEST['vote_minus'] : ''; switch ($route) { case '/good': action_good(); break; case '/bad': action_bad(); break; case '/ugly': action_ugly(); break; case '/about': action_about(); break; default: if (!empty($vote_plus) && !empty($vote_minus)) { sleep (1); do_post($vote_plus, $vote_minus); } action_main(); } 

Note the presence of two variables, $ vote_plus and $ vote_minus , which will be transmitted in a POST request when voting for one or another picture. The fact is that, knowing the name and path of the file is very easy to wind the voting bots, but we do not need it. Therefore, we will generate a pair of unique tokens for the voting page, one for each image. After the vote, the token will be deleted, making it impossible to reuse the vote.



Since in PHP before the release of version 7.0 the situation with crypto-secure functions is sad, I will help the rich capabilities of Tarantula to work with cryptography.



To begin with, in the action_main function, we initiate a random number generator with a crypto-safe (i.e. truly random) seed:

  $r = $tarantool->evaluate( "digest = require('digest') return (digest.urandom(4))" ); $seed = unpack('L', $r[0])[1]; srand($seed); 

The $ tarantool-> evaluate () function is used directly to start the Lua code without messing with the creation of a stored procedure. Then we call the create_random_vote () function twice , which will select a random element in space and create a URL for the image and tokens.





Two more functions were used here: $ tarantool-> call () for calling embedded procedures and $ tarantol-> insert () for inserting a new record.



Here is an example of a voting update procedure that updates the rating of a record:





For a complete list of Tarantool class methods, see the tarantool-php documentation.



Note the " op" => "=" parameter, which means that the field is replaced in an existing tuple. There is also a parameter +, - and some other operations. They perform a very important task. Usually, to replace a value in the database, we first read some field, then change it. To maintain data consistency, you have to block access to the table and use transactions. In Tarantula, due to its architecture, the update and upsert commands work atomically inside the server process without locking the database. It allows you to build damn fast systems!



The code of the index.php file at the time of this writing is under the spoiler:





It remains to make HTML-templates that will display data with pictures for voting on the site and rating pages.





Has anyone managed to meet in the life of a designer-maker who would perfectly format the HTML code?



From the above code, it is not difficult to guess that we gave out the same pair of tokens for the left and right pictures, just changed the vote_plus and vote_minus . (Usually, such phrases as the author emphasizes his intellectual superiority over readers). Thus, no matter what user clicks on the picture, it will get a plus, and its competitor will get a minus. The loser each time gets more and more minuses and falls lower and lower to hell. There he and the road HA - HA - HA!



The reader, who got to the end of the article, suffered the author's trolling and boiled more than once from his stupidity, he deserves a well-deserved reward. What could be more interesting than a real working example that you can poke and poke? Come and vote for Telegram stickers on ugly.begetan.me to find out which of them is the ugliest. Dig a mouse on our Russian fileload, compiled from NGNIX, PHP-FPM and Tarantool. And do not forget to show the link to your girlfriend, because you need a lot of votes to get a statistically reliable picture of the vote.