Nginx + Lua, flexible load balancing with session saving

When load balancing is an important issue - saving the client session. Especially if there is some kind of interactive backend behind the balancer. And even more so if you wanted to do A / B testing and flexibly adjust customer portions to different content. "Nginx plus" offers such opportunities, but what if you want cheap and fast?

The opportunity to expand the functionality of Nginx with the help of Lua comes to the rescue.

The algorithm is simple. At the first client request, we set a cookie for him, and at the next, depending on the value, we send to a specific backend. The cookies themselves are distributed by a suitable algorithm with the analysis of the necessary parameters.

As a powerful nginx harvester, you can use the OpenResty build, but for our needs this is redundant, so we will only collect the necessary functionality based on nginx 1.10.3 from the repository.

We will have a test subject:

 Debian jessie 4.9.0-0.bpo.1-amd64 Nginx 1.10.3 (nginx.org) libluajit-5.1-2 

Necessary components of the assembly:

 ngx_devel_kit-0.3.0 lua-nginx-module-0.10.8 lua-resty-core-0.1.11 lua-resty-lrucache-0.06 

Install packages to build the deb package:

 # cd /usr/src/ # aptitude install quilt debhelper libluajit-5.1-dev libluajit-5.1-2 # apt-get -t jessie source nginx 

The last command downloads the source codes of nginx from the configured repository. We use nginx: packages for Linux .

Download and unpack the current versions of the source code for the modules: ngx_devel_kit and lua-nginx-module

 # wget https://github.com/simpl/ngx_devel_kit/archive/v0.3.0.tar.gz # wget https://github.com/openresty/lua-nginx-module/archive/v0.10.8.tar.gz # tar -xf v0.3.0.tar.gz # tar -xf v0.10.8.tar.gz 

The first module is needed to build the desired second.

The rules for the build package for the deb-package at nginx-1.10.3/debian/rules config.status.nginx: config.env.nginx adding the config.status.nginx: config.env.nginx section to the list of parameters config.status.nginx: config.env.nginx :

 --add-module=/usr/src/ngx_devel_kit-0.3.0 --add-module=/usr/src/lua-nginx-module-0.10.8 

We assemble and install the resulting package:

 # cd nginx-1.10.3 && dpkg-buildpackage -us -uc -b && cd ../ # dpkg -i nginx_1.10.3-1~jessie_amd64.deb # aptitude hold nginx 

The last command will fix the installed package to avoid accidental updates.

In addition, we will need two more lua-libraries from the OpenResty project, which provide the Nginx API for Lua : lua-resty-core and lua-resty-lrucache . They are a set of *.lua files installed (by default) by the path /usr/local/lib/lua/ .

 # wget https://github.com/openresty/lua-resty-core/archive/v0.1.11.tar.gz # wget https://github.com/openresty/lua-resty-lrucache/archive/v0.06.tar.gz # tar -xf v0.1.11.tar.gz # tar -xf v0.06.tar.gz # cd lua-resty-core-0.1.11 && make install && cd ../ # cd lua-resty-lrucache-0.06 && make install && cd ../ 

The preparatory part is complete, proceed to setting up nginx. I will give a simplified configuration with comments of what is happening.

In our case, it was necessary to implement only content variants, therefore the balancer and backend will be on the same server and upstream will point to local addresses with 800x ports.
But implementation flexibility allows you to build any desired configuration. So in order.

In the http {} block, initialize lua.

 #     *.lua      lua_package_path "/usr/local/lib/lua/?.lua;;"; init_by_lua_block { --    --  ,     require "resty.core" collectgarbage("collect") -- just to collect any garbage } 

In the *_lua_block blocks, there is already a lua-code with its own syntax and functions.

The main server that accepts external requests.

 server { listen 80; server_name test.domain.local; location / { #   cookie "upid"    —     if ($cookie_upid = "") { #    nginx-,     ID  set $upstream_id ''; rewrite_by_lua_block { --          nginx- math.randomseed(ngx.time()) --    ,     ( ) math.random(100) local num = math.random(100) --  ,       20% / 80% if num > 20 then ngx.var.upstream_id = 1 ngx.ctx.upid = ngx.var.upstream_id else ngx.var.upstream_id = 2 ngx.ctx.upid = ngx.var.upstream_id end -- ID    nginx- "upstream_id"   "upid"  ngx.ctx  lua,          } #    "upid"    ID #     ,         (  ),    add_header Set-Cookie "upid=$upstream_id; Domain=$host; Path=/"; } #       ,   ID  ngx.ctx.upid   if ($cookie_upid != "") { rewrite_by_lua_block { ngx.ctx.upid = ngx.var.cookie_upid } } #      upstream- proxy_pass http://ab_test; } } 

An upstream block that, using lua, replaces the nginx built-in logic.

 upstream ab_test { # ,  nginx  .     server 127.0.0.1:8001; balancer_by_lua_block { local balancer = require "ngx.balancer" --    -- port  ,     ID  local host = "127.0.0.1" local port = 8000 + ngx.ctx.upid --   upstream     local ok, err = balancer.set_current_peer(host, port) if not ok then ngx.log(ngx.ERR, "failed to set the current peer: ", err) return ngx.exit(500) end --    ,  ,   ,      } } 

Well, a simple demo backend, which eventually come to customers.

 server { listen 127.0.0.1:8001; server_name test.domain.local; location / { root /var/www/html; index index.html; } } server { listen 127.0.0.1:8002; server_name test.domain.local; location / { root /var/www/html; index index2.html; } } 

When you run nginx-a with this configuration, a warning will be logged:

 use of lua-resty-core with LuaJIT 2.0 is not recommended; use LuaJIT 2.1+ instead while connecting to upstream 

Which can be removed by assembling and installing the required version. But also on 2.0 (libluajit-5.1-2) works.
Now, using a browser with developer tools, we can check the server’s work and set cookies.

Thus, we obtained the necessary flexibility for testing and statistics. And necessary for the correct operation of the backend saving client session. Well, just an interesting experience.

PS Similar problems can be solved by other methods, for example, using haproxy , which allows you to balance with sessions. Or, to split clients, use ngx_http_split_clients_module and use map to map one value to another.
But the given variant of distribution of clients and choice of backend allows to customize the system more flexibly. And if necessary, add a variety of logic to the work. And without rebuilding the current system.

Thanks for attention.