Comparative load testing of Lua-connectors for Tarantool from NGINX
Recently, on Habré, there are quite a few articles about Tarantool - a database and application server, which is used in Mail.Ru Group, Avito, Yota in various interesting projects. And so, I thought - and why are we worse? Let's try too.
By virtue of my professional deformation I will consider the following case:
How to approach this task?
Let's put in front of the resource a gateway that will check the rights of users, and depending on the results of the test, allow or not allow the user to the resource. User access rights will be stored in Tarantool. It has master-master replication, and if we need to build a cluster of gateways, it will come in handy. As a basis for the gateway, we will use NGINX (well, don’t write the Web server itself ...).
')
It is necessary to add “intelligence” to NGINX so that it understands where the user can go, and where it is impossible. For this one could use ngx_http_auth_request_module , but it is not clear how to combine this with Tarantool. Let's follow the example of OpenResty, and use it to intellectualize our Lua gateway, namely lua-nginx-module .
To access Tarantool from within NGINX, we need the appropriate driver, or “connector” for the selected language. The authors themselves Tarantool write that if you needed to have a connector to Tarantool from Lua - then you have something wrong with the architecture. But in the case of the NGINX + Lua bundle, this can be justified.
Googling shows that there are already three candidates in nature:
What to choose? Need to test. That and do.
From the load generator, requests are transmitted to the gateway in a protected TLS form (do not forget about the prof. Deformation). Next, NGINX removes TLS, and sends them to the protected resource in the form of regular HTTP.
(two identical cars)
Nothing out of the ordinary - just an empty gif.
Pay attention to the line:
this is where we check user rights.
It's simple:
1. Extract the authorization cookie;
2. Parsim request to understand which resource the user is accessing;
3. We transfer the obtained values of Tarantool so that it makes a decision whether to let the user or not.
4. We process the answer Tarantool
5. Depending on the answer, let the user go, or say “Access Denied”.
For simplicity, we will leave work with access rights outside the article, and we will always let the user access the resource.
Since we still ignore the authorization cookie, for simplicity we will not consider the process of its receipt and formation.
For testing, we will use the wrk utility. It has worked well in load testing and, in addition to TLS, supports Lua scripts (although we will not use them now). Of the features, wrk does not have a disconnected TLS Session Resumption, so the gateway's CPU will not be spent on permanent TLS handshakes. In order to check the performance of access permissions per second, and not throughput, we will request a file of the minimum size from the protected resource - an empty GIF that occupies 43 bytes.
Let's start testing.
Does not work at all. If you use it according to the documentation, then
An error:
We will not understand.
First - a little. Only 125 requests per second.
Secondly - more than half of the answers - not the expected 200, but something else. What is this? The answer is in error_log NGIXN:
Something goes wrong - either Tarantool rejects the connections, or NGINX cannot digest them.
But let's try further.
Wow! 3500 requests per second, and not a single error!
And in error_log NGINX - silence.
Despite the venerable age, and some shortcomings, Candidate 3 (lua-resty-tarantool) is clearly not just a leader, but the only use in production. And once again we were convinced of the need to test various use cases before making decisions about using or not using one or another technology in real projects.
By virtue of my professional deformation I will consider the following case:
- There is a Web-resource, access to which we want to restrict;
- The resource itself cannot be changed or is highly undesirable.
How to approach this task?
Let's put in front of the resource a gateway that will check the rights of users, and depending on the results of the test, allow or not allow the user to the resource. User access rights will be stored in Tarantool. It has master-master replication, and if we need to build a cluster of gateways, it will come in handy. As a basis for the gateway, we will use NGINX (well, don’t write the Web server itself ...).
')
It is necessary to add “intelligence” to NGINX so that it understands where the user can go, and where it is impossible. For this one could use ngx_http_auth_request_module , but it is not clear how to combine this with Tarantool. Let's follow the example of OpenResty, and use it to intellectualize our Lua gateway, namely lua-nginx-module .
To access Tarantool from within NGINX, we need the appropriate driver, or “connector” for the selected language. The authors themselves Tarantool write that if you needed to have a connector to Tarantool from Lua - then you have something wrong with the architecture. But in the case of the NGINX + Lua bundle, this can be justified.
Googling shows that there are already three candidates in nature:
- github.com/ziontab/lua-nginx-tarantool - Implemented on nginx cosockets. The truth has not been updated for a long time.
- github.com/tarantool/tarantool-lua - the official driver from the Tarantool developers. It is a modified fork of the first candidate. In addition to nginx cosockets supports ordinary Lua sockets.
- github.com/perusio/lua-resty-tarantool - For two years now without commits, there is no support for the semantics of calls to Tarantool 1.7 procedures.
What to choose? Need to test. That and do.
Test bench:
From the load generator, requests are transmitted to the gateway in a protected TLS form (do not forget about the prof. Deformation). Next, NGINX removes TLS, and sends them to the protected resource in the form of regular HTTP.
Characteristics of test machines
Load machine and protected resource
(two identical cars)
| CPU | 2xIntel Xeon E5 2680 @ 2.70GHz Sandy Bridge-EP / EX 32nm Technology 8 Cores / 16 threads |
| Ram | 32.0GB DDR3 @ 799MHz (11-11-11-28) |
| MB | Supermicro X9DR3-F |
| Disk | 223GB OCZ-VERTEX3 (SSD) |
| OS | Debian 8.9 x64 (kernel 3.16.39-1) |
| Nginx | 1.12.1 |
| wrk | 4.0.2-dirty [epoll] + GOST TLS patches |
Gateway
| CPU | 1 vCPU |
| Ram | 8 Gb |
| Platform | VMWare Workstation 12.5 |
| Host CPU | Intel Core i5 7600K 3.8 GHz |
| Host ram | 16 Gb |
| Host os | Windows 10 x64 |
| Nginx | 1.12.1 |
| lua-nginx-module | Latest master branch |
Config NGINX protected resource
Nothing out of the ordinary - just an empty gif.
user nginx; worker_processes 32; error_log /var/log/ngate/nginx/error.log warn; pid /var/run/nginx.pid; worker_rlimit_nofile 65535; events { worker_connections 8192; } http { access_log /var/log/ngate/nginx/access.log main; keepalive_timeout 65; server { listen 80; server_name fast-ipsec2-db8; location / { root /var/www; index index.html index.htm; } location = /ff/empty_gif.gif { empty_gif; } } # end server } NGINX gateway config:
worker_processes 1; error_log /var/log/nginx/error.log warn; worker_rlimit_nofile 65535; events { worker_connections 8192; } http { include /etc/opt/nginx/mime.types; default_type text/html; sendfile on; keepalive_timeout 65; autoindex off; server_tokens off; lua_package_path '?.lua;/opt/lua/?.lua;'; # HTTPS server server { listen 443 ssl; server_name perf-test-1; ssl_certificate www.example.com.crt; ssl_certificate_key www.example.com.key; ssl_protocols TLSv1; ssl_ciphers HIGH:!aNULL:!MD5; if ($request_method !~ ^(GET|HEAD|POST)$ ) { return 444; } # Local Tarantool Node set $ng_local_tnt_addr '127.0.0.1'; set $ng_local_tnt_port 3320; # ff location /ff/ { proxy_pass http://fast-ipsec2-db8/ff/; access_by_lua_file /opt/lua/res_access.lua; } } # end server perf-test-1 } # end http Pay attention to the line:
access_by_lua_file /opt/lua/res_access.lua; this is where we check user rights.
Code /opt/lua/res_access.lua
It's simple:
1. Extract the authorization cookie;
2. Parsim request to understand which resource the user is accessing;
3. We transfer the obtained values of Tarantool so that it makes a decision whether to let the user or not.
4. We process the answer Tarantool
5. Depending on the answer, let the user go, or say “Access Denied”.
For simplicity, we will leave work with access rights outside the article, and we will always let the user access the resource.
local auth_cookie_value = ngx.var.cookie_nginxauth if auth_cookie_value == nil then ngx.log(ngx.WARN, "Authentication cookie not provided.") ngx.exit(ngx.HTTP_NOT_FOUND) end local uri_root_regex = "(\\/[a-zA-Z0-9\\-\\._]+\\/)" local m, err = ngx.re.match(ngx.var.uri, uri_root_regex, "ai") if err then ngx.log(ngx.ERR, "Error in regexp: ", err) ngx.exit(ngx.HTTP_NOT_FOUND) end if m == nil then ngx.log(ngx.ERR, "Regexp returned nil value.") ngx.exit(ngx.HTTP_NOT_FOUND) end local uri_root = m[0] if uri_root == nil then ngx.log(ngx.ERR, "error in regexp") ngx.exit(ngx.HTTP_NOT_FOUND) end local tnt = require 'resty.tarantool' #local tnt = require 'tarantool-lua.tarantool' local tar, err = tnt:new({ host = ngx.var.ng_local_tnt_addr, port = ngx.var.ng_local_tnt_port, --Default value 2000 socket_timeout = 500, # connect_now = false, }) if not tar:connect() then ngx.log(ngx.ERR, "TNT connection failed.") ngx.exit(ngx.HTTP_NOT_FOUND) end local res, err = tar:call('check_access', {auth_cookie_value, uri_root}) if not tar:set_keepalive() then ngx.log(ngx.WARN, "TNT connection not set as keep-alive.") end if not res then ngx.log(ngx.ERR, "TNT call failed: " .. err) ngx.exit(ngx.HTTP_NOT_FOUND) end if res[1] ~= nil and res[1][1] == true then -- Access granted ngx.log(ngx.INFO, "Resource access granted: " .. uri_root) return else ngx.log(ngx.ERR, "Resource access denied: " .. uri_root) ngx.exit(ngx.HTTP_FORBIDDEN) end Tarantool stored procedure code
Since we still ignore the authorization cookie, for simplicity we will not consider the process of its receipt and formation.
local strict = require('strict') strict.on() function check_access(session_id, resource_name) if session_id == nil or resource_name == nil then return false end log.info('Access to resource ' .. resource_name .. ' granted.') return true end Testing method
For testing, we will use the wrk utility. It has worked well in load testing and, in addition to TLS, supports Lua scripts (although we will not use them now). Of the features, wrk does not have a disconnected TLS Session Resumption, so the gateway's CPU will not be spent on permanent TLS handshakes. In order to check the performance of access permissions per second, and not throughput, we will request a file of the minimum size from the protected resource - an empty GIF that occupies 43 bytes.
Let's start testing.
Candidate 1 (lua-nginx-tarantool):
Does not work at all. If you use it according to the documentation, then
local tnt = require 'lua-nginx-tarantool.tarantool' local tar, err = tnt:new({...}) An error:
runtime error: /opt/lua/res_access.lua:33: attempt to index local 'tnt' (a boolean value) We will not understand.
Candidate 2 (tarantool-lua):
./wrk -t32 -c32 -d30s --latency --timeout 10s -H "Host: perf-test-1" -H "Cookie: nginxauth=XXX " https://192.168.85.159/ff/empty_gif.gif Thread Stats Avg Stdev Max +/- Stdev Latency 252.12ms 248.32ms 514.22ms 31.13% Req/Sec 4.55 5.54 60.00 95.77% Latency Distribution 50% 21.75ms 75% 502.22ms 90% 503.61ms 99% 507.63ms 3767 requests in 30.04s, 1.33MB read Non-2xx or 3xx responses: 1868 Requests/sec: 125.40 Transfer/sec: 45.49KB First - a little. Only 125 requests per second.
Secondly - more than half of the answers - not the expected 200, but something else. What is this? The answer is in error_log NGIXN:
[error] 11856#0: *23410 [lua] res_access.lua:42: TNT connection failed. Something goes wrong - either Tarantool rejects the connections, or NGINX cannot digest them.
But let's try further.
Candidate 3 (lua-resty-tarantool):
./wrk -t32 -c32 -d30s --latency --timeout 10s -H "Host: perf-test-1" -H "Cookie: nginxauth=XXX " https://192.168.85.159/ff/empty_gif.gif Thread Stats Avg Stdev Max +/- Stdev Latency 9.03ms 1.13ms 28.79ms 79.90% Req/Sec 110.13 10.43 131.00 75.49% Latency Distribution 50% 8.63ms 75% 9.46ms 90% 10.64ms 99% 11.81ms 105344 requests in 30.03s, 43.40MB read Requests/sec: 3508.50 Transfer/sec: 1.45MB Wow! 3500 requests per second, and not a single error!
And in error_log NGINX - silence.
findings
Despite the venerable age, and some shortcomings, Candidate 3 (lua-resty-tarantool) is clearly not just a leader, but the only use in production. And once again we were convinced of the need to test various use cases before making decisions about using or not using one or another technology in real projects.
Source: https://habr.com/ru/post/336252/
All Articles