Applications for Tarantool. Part 3. Testing and launch

The application for Tarantool is, in fact, a set of stored procedures used as an API. Data is processed on the storage side, which can significantly improve performance. However, supporting stored procedures can turn into a nightmare.

Can. But not today.

Today we will look at the quality assurance of the application. In particular, let's talk about testing, figure out how to run in production, how to use connectors, and also talk about the intricacies of data schema migration.

Content of the cycle “Applications for Tarantool”

- Part 1. Stored procedures
- Part 2. OAuth2 authentication
- Part 3. Testing and running.

Let me remind you that all the examples in this series of articles are based on the application for authorization of users tarantool-authman .

Testing

Testing is only one way to improve quality.

In the case of stored procedures, functional tests are a good way to support the current API. For testing, the Tap (Test Anything Protocol) module built into Tarantool is used. It includes several methods for basic checks and comparisons of Lua objects. Let's test two cases of user registration auth.registration(email) .

 local case = {} local tap = require('tap') local response = require('authman.response') local error = require('authman.error') local db = require('authman.db') local config = require('test.config') --     local auth = require('authman').api(config) local test = tap.test('registration_test') --  space     function case.before() db.truncate_spaces() end function case.after() end function test_registration_succes() local ok, code ok, code = auth.registration('test@test.ru') -- ,    ,     —  test:is(ok, true, 'test_registration_succes user created') test:isstring(code, 'test_registration_succes code returned') end function test_registration_user_already_exists() local ok, code, got, expected, user ok, code = auth.registration('test@test.ru') --    API,      ok, user = auth.complete_registration('test@test.ru', code, 'password') -- ,      --    ,         Lua- got = {auth.registration('test@test.ru'), } expected = {response.error(error.USER_ALREADY_EXISTS), } -- is_deeply       Lua- test:is_deeply(got, expected, 'test_registration_user_already_active') end case.tests = { test_registration_succes, test_registration_user_already_exists } return case 

Now we will write a small script to run the tests. Do not forget to also take into account the before and after methods, which will simplify testing.

 --  ,   Tarantool box.cfg { listen = 3331, } local TEST_CASES = { 'test.case.registration', } function run() for case_index = 1, #TEST_CASES do local case = require(TEST_CASES[case_index]) for test_index = 1, #case.tests do --    case.before() case.tests[test_index]() case.after() end end end run() 

The result of running tests:

 $ tarantool test/authman.test.lua ... TAP version 13 ok — test_registration_succes user created ok — test_registration_succes code returned ok — test_registration_user_already_active 

Instance running

The application is written and tested, which means it's time to deploy it in a production-environment.

To manage Tarantool instances, the tarantoolctl utility is used . At first we will create instans with the application. Do not forget to add the user and give him rights.

 box.cfg { listen = 3331; } local function bootstrap() box.schema.user.create('my_user', {password = '123'}) box.schema.user.grant('my_user', 'read,write,execute', 'universe') end box.once('init_user', bootstrap) config = { --   } --   auth  auth = require('auth').api(config) 

Now create a symbolic link to this file from the instances.enabled directory:

 sudo ln -s /etc/tarantool/instances.available/auth.lua /etc/tarantool/instances.enabled/auth.lua 

Run tarantoolctl and verify that we can connect to Tarantool and use application methods:

 $ tarantoolctl start auth $ tarantoolctl enter auth connected to unix/:/var/run/tarantool/auth.control unix/:/var/run/tarantool/auth.control> ok, user = auth.registration('ivanov@mail.ru') unix/:/var/run/tarantool/auth.control> ok, user = auth.complete_registration('ivanov@mail.ru', user.code, '123') unix/:/var/run/tarantool/auth.control> user --- - is_active: true email: ivanov@mail.ru id: 8cd27d26-3974-43d6-a2b2-87202664753d ... 

If something went wrong, you should look at the logs /var/log/tarantool/auth.lua . More information about administering the Tarantool server can be found in the documentation .

Connector

One of the advantages of stored procedures is the provision of a common interface for services written in different programming languages.

Consider an example of asynchronous use of Tarantool using a Python connector .

 import asyncio import asynctnt async def create_user(): tnt_connection = asynctnt.Connection( host='127.0.0.1', port='3367', username='my_user', password='123' ) #    user_data = await tnt_connection.call( 'auth.registrtion', ['exaple@mail.ru', ] ) await tnt_connection.disconnect() #    loop = asyncio.get_event_loop() loop.run_until_complete(create_user()) 

It is worth noting that you can also interact with Tarantool from another Tarantool instance using the built-in net.box module. Read more about it in the documentation .

Data migration

A common task when an application is already running in combat is changing the data schema. For example, the user is required to add the gender field.

To do this, create a script that changes all the data in space auth_user . The script itself can be added to the initialization call of the application, and the box.once function box.once restrict the repeated call.

 local fiber = require('fiber') local function migrations() --      box.once('20171101_add_gender', function () local counter = 0 --  space        --       for _, tuple in box.space.auth_user:pairs( nil, {iterator=box.index.ALL} ) do local user_tuple = tuple:totable() --      user_tuple[4] = get_user_gender() box.space.auth_user:replace(user_tuple) counter = counter + 1 end end) end migrations() 

Although the migration can take a long time, the main thread of execution is not blocked. This is due to the fact that the replace method implicitly calls yield . In order to explicitly release the execution thread, you need to call the fiber.sleep(0) method. Read more about this in the documentation .

What's next?

Tarantool does not stand still and is actively developing.

Outside this training cycle of articles, there are several important issues that you should pay attention to when writing your own services:

• Sharding and replication. Using the vshard module .
• SQL syntax support in Tarantool. Optimize stored procedures using SQL.
• Review of the Vinyl disk engine. Comparison with Memtx, or "What should I do if the RAM has run out?"
• Optimization and profiling of stored procedures written in C.

I remind you that all code examples are taken from the tarantool-authman application . It is also evolving, and can now be used as an Oauth2-server.

If you have any questions beyond the cycle of articles, you can ask them in telegram-chat . A responsive Tarantool community will most likely be able to help you.

Thanks to everyone who waited for the third part.

See you again!

The Tarantool community holds the first in-memory computing conference on June 21 in Moscow. Reports are still accepted, call for papers is open.