Tarantool application: stored procedures

Translation of an article with DZone. Original

I want to share my experience in creating applications for Tarantool, and today we will talk about installing this database, storing data and accessing them, as well as writing stored procedures.

Tarantool is a NoSQL / NewSQL database that stores data in RAM, but can use a disk and ensure consistency through a carefully designed mechanism called the write-ahead log (WAL). Tarantool also boasts a built-in LuaJIT compiler (JIT - just-in-time), which allows you to execute Lua-code.

The first steps

We will consider the creation of a Tarantool application that implements an API for registering and authenticating users. Its features:

• Registration and authentication by mail in three stages: account creation, registration confirmation and password setting.
• Registration using social network credentials (FB, Google+, VKontakte, etc.).
• Password recovery.

For an example of a stored procedure for Tarantool, we turn to the first stage, or rather to obtain a registration confirmation code. You can go to the GitHub repository and perform all actions in the course of the story.

Install Tarantool

The network has detailed installation instructions for various operating systems. For example, to install Tarantool under Ubuntu, paste it into the console and execute this script:

curl http://download.tarantool.org/tarantool/1.9/gpgkey | sudo apt-key add - release=`lsb_release -c -s` sudo apt-get -y install apt-transport-https sudo rm -f /etc/apt/sources.list.d/*tarantool*.list sudo tee /etc/apt/sources.list.d/tarantool_1_9.list <<- EOF deb http://download.tarantool.org/tarantool/1.9/ubuntu/ $release main deb-src http://download.tarantool.org/tarantool/1.9/ubuntu/ $release main EOF sudo apt-get update sudo apt-get -y install tarantool 

Verify the success of the installation by typing tarantool and logging in to the interactive administrator console.

 $ tarantool version 1.9.0-4-g195d446 type 'help' for interactive help tarantool> 

Here you can already try programming on Lua. If you are not familiar with this language, then here is a quick start guide: http://tylerneylon.com/a/learn-lua .

Registration by mail

Now we will write our first script to create a space in which all users will be stored. It is similar to a table in a relational database. The data itself is stored in tuples (arrays containing records). Each space must have one primary index and may have several secondary indexes. Indices can be defined by one or several fields. Here is a space map of our authentication service:

We use two types of indices: HASH and TREE . The HASH index allows you to search for tuples using a full match of the primary key, which must be unique. The TREE index supports non-unique keys, allows you to search at the beginning of a composite index and organize the sorting of keys, since their values ​​are ordered within the index.

The session space contains a special key ( session_secret ) used to sign session cookies. Storing session keys allows you to log out users on the server side if necessary. The session also has an optional reference to the social space. This is needed to check the sessions of those users who log on to the social network credentials (we check the validity of the stored OAuth 2 token).

We write application

Before you start writing the application itself, let's take a look at the project structure:

 tarantool-authman ├── authman │ ├── model │ │ ├── password.lua │ │ ├── password_token.lua │ │ ├── session.lua │ │ ├── social.lua │ │ └── user.lua │ ├── utils │ │ ├── http.lua │ │ └── utils.lua │ ├── db.lua │ ├── error.lua │ ├── init.lua │ ├── response.lua │ └── validator.lua └── test ├── case │ ├── auth.lua │ └── registration.lua ├── authman.test.lua └── config.lua 

The paths defined in the package.path variable are used to import Lua packages. In our case, the packages are imported relative to the current tarantool-authman . But if needed, the import paths can be easily extended:

 -- Prepending a new path with the highest priority package.path = “/some/other/path/?.lua;” .. package.path 

Before creating the first space, let's put all the necessary constants in separate models. You need to give a name to each space and index. You also need to determine the order of the fields in the tuple. For example, this is the model authman/model/user.lua :

 -- Our package is a Lua table local user = {} -- The package has the only function — model — that returns a table -- with the model's fields and methods -- The function receives configuration in the form of a Lua table function user.model(config) local model = {} -- Space and index names model.SPACE_NAME = 'auth_user' model.PRIMARY_INDEX = 'primary' model.EMAIL_INDEX = 'email_index' -- Assigning numbers to tuple fields -- Note thatLua uses one-based indexing! model.ID = 1 model.EMAIL = 2 model.TYPE = 3 model.IS_ACTIVE = 4 -- User types: registered via email or with social network -- credentials model.COMMON_TYPE = 1 model.SOCIAL_TYPE = 2 return model end -- Returning the package return user 

When processing users, we will need two indexes: unique by ID and non-unique by mail address. When two different users are registered with social network credentials, they can specify the same addresses or not specify them at all. And for users who register in the usual way, the application will check the uniqueness of postal addresses.

The authman/db.lua contains a method for creating spaces:

 local db = {} -- Importing the package and calling the model function -- The config parameter is assigned a nil (empty) value local user = require('authman.model.user').model() -- The db package's method for creating spaces and indexes function db.create_database() local user_space = box.schema.space.create(user.SPACE_NAME, { if_not_exists = true }) user_space:create_index(user.PRIMARY_INDEX, { type = 'hash', parts = {user.ID, 'string'}, if_not_exists = true }) user_space:create_index(user.EMAIL_INDEX, { type = 'tree', unique = false, parts = {user.EMAIL, 'string', user.TYPE, 'unsigned'}, if_not_exists = true }) end return db 

The UUID will act as the user ID, and we will use the HASH index for a full match search. The index for searching by mail will consist of two parts: ( user.EMAIL, 'string' ) - the user's email address, ( user.TYPE, 'unsigned' ) - the type of user. I recall that the types are determined a little earlier in the model. The composite index allows you to search not only in all fields, but also in the first part of the index. So we can only search by mail address (without user type).

Log in to the admin console and use the authman/db.lua .

 $ tarantool version 1.9.0-4-g195d446 type 'help' for interactive help tarantool> db = require('authman.db') tarantool> box.cfg({}) tarantool> db.create_database() 

Great, we just created the first space. Don't forget: before calling box.schema.space.create , you need to configure and start the server using the box.cfg method. Now you can perform some simple actions inside the created space:

 -- Creating users tarantool> box.space.auth_user:insert({'user_id_1', 'example_1@mail.ru', 1}) — - - ['user_id_1', 'example_1@mail.ru', 1] … tarantool> box.space.auth_user:insert({'user_id_2', 'example_2@mail.ru', 1}) — - - ['user_id_2', 'example_2@mail.ru', 1] … -- Getting a Lua table (array) with all the users tarantool> box.space.auth_user:select() — - - — ['user_id_2', 'example_2@mail.ru', 1] — ['user_id_1', 'example_1@mail.ru', 1] … -- Getting a user by the primary key tarantool> box.space.auth_user:get({'user_id_1'}) — - - ['user_id_1', 'example_1@mail.ru', 1] … -- Getting a user by the composite key tarantool> box.space.auth_user.index.email_index:select({'example_2@mail.ru', 1}) — - - — ['user_id_2', 'example_2@mail.ru', 1] … -- Changing the data in the second field tarantool> box.space.auth_user:update('user_id_1', {{'=', 2, 'new_email@mail.ru'}, }) — - - ['user_id_1', 'new_email@mail.ru', 1] … 

Unique indexes do not allow non-unique values. If you want to create records that may already be in space, use the operation upsert (update / insert). A complete list of available methods is given in the official documentation .

Let's expand the user model with the option of registering users:

 function model.get_space() return box.space[model.SPACE_NAME] end function model.get_by_email(email, type) if validator.not_empty_string(email) then return model.get_space().index[model.EMAIL_INDEX]:select({email, type})[1] end end -- Creating a user -- Fields that are not part of the unique index are not mandatory function model.create(user_tuple) local user_id = uuid.str() local email = validator.string(user_tuple[model.EMAIL]) and user_tuple[model.EMAIL] or '' return model.get_space():insert{ user_id, email, user_tuple[model.TYPE], user_tuple[model.IS_ACTIVE], user_tuple[model.PROFILE] } end -- Generating a confirmation code sent via email and used for -- account activation -- Usually, this code is embedded into a link as a GET parameter -- activation_secret — one of the configurable parameters when -- initializing the application function model.generate_activation_code(user_id) return digest.md5_hex(string.format('%s.%s', config.activation_secret, user_id)) end 

The following code uses two standard packages Tarantool - uuid and digest - and one created by the user - validator . But first you need to import them:

 -- standard Tarantool packages local digest = require('digest') local uuid = require('uuid') -- Our application's package (handles data validation) local validator = require('authman.validator') 

When defining variables, we use the local operator, which limits their scope to the current block. If you do not do this, the variables will be global, and we need to avoid this because of possible name conflicts.

Now we will create the main authman/init.lua , which will store all API methods:

 local auth = {} local response = require('authman.response') local error = require('authman.error') local validator = require('authman.validator') local db = require('authman.db') local utils = require('authman.utils.utils') -- The package returns the only function — api — that configures and -- returns the application function auth.api(config) local api = {} -- The validator package contains checks for various value types -- This package sets the default values as well config = validator.config(config) -- Importing the models for working with data local user = require('authman.model.user').model(config) -- Creating a space db.create_database() -- The api method creates a non-active user with a specified email -- address function api.registration(email) -- Preprocessing the email address — making it all lowercase email = utils.lower(email) if not validator.email(email) then return response.error(error.INVALID_PARAMS) end -- Checking if a user already exists with a given email -- address local user_tuple = user.get_by_email(email, user.COMMON_TYPE) if user_tuple ~= nil then if user_tuple[user.IS_ACTIVE] then return response.error(error.USER_ALREADY_EXISTS) else local code = user.generate_activation_code(user_tuple[user.ID]) return response.ok(code) end end -- Writing data to the space user_tuple = user.create({ [user.EMAIL] = email, [user.TYPE] = user.COMMON_TYPE, [user.IS_ACTIVE] = false, }) local code = user.generate_activation_code(user_tuple[user.ID]) return response.ok(code) end return api end return auth 

Fine! Now users can create accounts.

 tarantool> auth = require('authman').api(config) -- Using the api to get a registration confirmation code tarantool> ok, code = auth.registration('example@mail.ru') -- This code needs to be sent to a user's email address so that they -- can activate their account tarantool> code 022c1ff1f0b171e51cb6c6e32aefd6ab 

To be continued