Nginx on steroids - expanding functionality with LUA

To ensure the operation of all our external products, we use the popular nginx. It is fast and reliable. There are almost no problems with it. Our products are also constantly evolving, new services are emerging, new functionality is being added, and old ones are expanding. The audience and the load is only growing. Now we want to talk about how we accelerated development, increased the performance quite well and simplified the addition of this new functionality to our services, while maintaining the availability and resiliency of the affected applications. It will be about the concept of “nginx as web application”.
Namely, about third-party modules (mainly LUA), allowing you to do completely magical things quickly and reliably.

Problems and solution

The main idea is quite simple. Take the following factors:
- the complexity of the application logic,
- the number of application components,
- the size of the audience.
From a certain point it becomes quite difficult to keep the application responsive and fast, sometimes even workable. The product becomes multi-component, geographically distributed. And more and more people use it. At the same time, there are business requirements for responsiveness and resiliency, which must first be respected.
There are several ways to solve this problem. Everything can be broken and altered to other technologies. Of course, this option works, but we didn’t like it very much and we decided to redo it gradually. The basis was the openresty build (nginx + LUA). Why lua. Without the help of cgi, fastcgi and other cgi, right in the nginx configuration file you can script powerful, beautiful and fast functionality. Everything works asynchronously. And not only with clients, but also with backends. At the same time, it does not interfere with the event loop of the web server, without callbacks, fully using the existing nginx functionality.

Currently the following backends are available:
- Redis
- memcache
- MySQL
- PostgreSQL
In addition, you can connect more modules for use, such as RabbitMQ and ZeroMQ .
It works pretty fast. Anyway, faster than php-fpm))
')
The logical question is, why not rewrite everything at all in C? Writing to LUA is much easier and faster. And we are immediately spared from the problems associated with asynchrony and the nginx event loop.

Examples Ideas

We, as usual, will not give the full code, only the main parts. These all things were made in php before.

1. This part was invented and made by our colleague AotD . There is a repository of pictures. They should be shown to users, and it is desirable to perform some operations, for example, resize. Pictures we store in ceph, this is an analogue of Amazon S3. For image processing used ImageMagick. There is a cache directory on the resizer; the processed images are added to it.
Parsing the user's request, defining the image, the resolution he needs and going to ceph, then processing and displaying it on the fly.
serve_image.lua

require "config" local function return_not_found(msg) ngx.status = ngx.HTTP_NOT_FOUND if msg then ngx.header["X-Message"] = msg end ngx.exit(0) end local name, size, ext = ngx.var.name, ngx.var.size, ngx.var.ext if not size or size == '' then return_not_found() end if not image_scales[size] then return_not_found('Unexpected image scale') end local cache_dir = static_storage_path .. '/' .. ngx.var.first .. '/' .. ngx.var.second .. '/' local original_fname = cache_dir .. name .. ext local dest_fname = cache_dir .. name .. size .. ext -- make sure the file exists local file = io.open(original_fname) if not file then -- download file contents from ceph ngx.req.read_body() local data = ngx.location.capture("/ceph_loader", {vars = { name = name .. ext }}) if data.status == ngx.HTTP_OK and data.body:len()>0 then os.execute( "mkdir -p " .. cache_dir ) local original = io.open(original_fname, "w") original:write(data.body) original:close() else return_not_found('Original returned ' .. data.status) end end local magick = require("imagick") magick.thumb(original_fname, image_scales[size], dest_fname) ngx.exec("@after_resize") 

We connect imagic.lua binding . Must be available LuaJIT .

nginx_partial_resizer.conf.template

 # Old images location ~ ^/(small|small_wide|medium|big|mobile|scaled|original|iphone_(preview|retina_preview|big|retina_big|small|retina_small))_ { rewrite /([^/]+)$ /__CEPH_BUCKET__/$1 break; proxy_pass __UPSTREAM__; } # Try get image from ceph, then from local cache, then from scaled by lua original # If image test.png is original, when user wants test_30x30.png: # 1) Try get it from ceph, if not exists # 2) Try get it from /cache/t/es/test_30x30.ong, if not exists # 3) Resize original test.png and put it in /cache/t/es/test_30x30.ong location ~ ^/(?<name>(?<first>.)(?<second>..)[^_]+)((?<size>_[^.]+)|)(?<ext>\.[a-zA-Z]*)$ { proxy_intercept_errors on; rewrite /([^/]+)$ /__CEPH_BUCKET__/$1 break; proxy_pass __UPSTREAM__; error_page 404 403 = @local; } # Helper failover location for upper command cause you can't write # try_files __UPSTREAM__ /cache/$uri @resizer =404; location @local { try_files /cache/$first/$second/$name$size$ext @resize; } # If scaled file not found in local cache resize it with lua magic! location @resize { # lua_code_cache off; content_by_lua_file "__APP_DIR__/lua/serve_image.lua"; } # serve scaled file, invoked in @resizer serve_image.lua location @after_resize { try_files /cache/$first/$second/$name$size$ext =404; } # used in @resizer serve_image.lua to download original image # $name contains original image file name location =/ceph_loader { internal; rewrite ^(.+)$ /__CEPH_BUCKET__/$name break; proxy_set_header Cache-Control no-cache; proxy_set_header If-Modified-Since ""; proxy_set_header If-None-Match ""; proxy_pass __UPSTREAM__; } location =/favicon.ico { return 404; } location =/robots.txt {} 

2. Firewall for API. Validation of the request, customer identification, control of rps and a barrier for those who do not need us.
Firewall.lua

 module(..., package.seeall); local function ban(type, element) CStorage.banPermanent:set(type .. '__' .. element, 1); ngx.location.capture('/postgres_ban', { ['vars'] = { ['type'] = type, ['value'] = element} }); end local function checkBanned(apiKey) -- init search criteria local searchCriteria = {}; searchCriteria['key'] = apiKey; if ngx.var.remote_addr then searchCriteria['ip'] = ngx.var.remote_addr; end; -- search in ban lists for type, item in pairs(searchCriteria) do local storageKey = type .. '__' .. item; if CStorage.banPermanent:get(storageKey) then ngx.exit(444); elseif CStorage.banTmp:get(storageKey) then -- calculate rps and check is our client still bad boy 8-) local rps = CStorage.RPS:incr(storageKey, 1); if not(rps) then CStorage.RPS:set(storageKey, 1, 1); rps=1; end; if rps then if rps > config.app_params['ban_params']['rps_for_ip_to_permanent_ban'] then CStorage.RPS:delete(storageKey); ban(type, item); ngx.exit(444); elseif config.app_params['ban_params']['rps_for_ip_to_tmp_ban'] > 0 and rps == config.app_params['ban_params']['rps_for_ip_to_tmp_ban'] then local attemptsCount = CStorage.banTmp:incr(storageKey, 1) - 1; if attemptsCount > config.app_params['ban_params']['tmp_ban']['max_attempt_to_exceed_rps'] then -- permanent ban CStorage.banTmp:delete(storageKey); ban(type, item); end; end; end; ngx.exit(444); end; end; end; local function checkTemporaryBlocked(apiKey) local blockedData = CStorage.tmpBlockedDemoKeys:get(apiKey); if blockedData then --storage.tmpBlockedDemoKeys:incr(apiKey, 1); -- think about it. return CApiException.throw('tmpDemoBlocked'); end; end; local function checkRPS(apiKey) local rps = nil; -- check rps for IP and ban it if it's needed if ngx.var.remote_addr then local ip = 'ip__' .. tostring(ngx.var.remote_addr); rps = CStorage.RPS:incr(ip, 1); if not(rps) then CStorage.RPS:set(ip, 1, 1); rps = 1; end; if rps > config.app_params['ban_params']['rps_for_ip_to_permanent_ban'] then ban('ip', tostring(ngx.var.remote_addr)); ngx.exit(444); elseif config.app_params['ban_params']['rps_for_ip_to_tmp_ban'] > 0 and rps > config.app_params['ban_params']['rps_for_ip_to_tmp_ban'] then CStorage.banTmp:set(ip, 1, config.app_params['ban_params']['tmp_ban']['time']); ngx.exit(444); end; end; local apiKey_key_storage = 'key_' .. apiKey['key']; -- check rps for key rps = CStorage.RPS:incr(apiKey_key_storage, 1); if not(rps) then CStorage.RPS:set(apiKey_key_storage, 1, 1); rps = 1; end; if apiKey['max_rps'] and rps > tonumber(apiKey['max_rps']) then if apiKey['mode'] == 'demo' then CApiKey.blockTemporary(apiKey['key']); return CApiException.throw('tmpDemoBlocked'); else CApiKey.block(apiKey['key']); return CApiException.throw('blocked'); end; end; -- similar check requests per period (RPP) for key if apiKey['max_request_count_per_period'] and apiKey['period_length'] then local rpp = CStorage.RPP:incr(apiKey_key_storage, 1); if not(rpp) then CStorage.RPP:set(apiKey_key_storage, 1, tonumber(apiKey['period_length'])); rpp = 1; end; if rpp > tonumber(apiKey['max_request_count_per_period']) then if apiKey['mode'] == 'demo' then CApiKey.blockTemporary(apiKey['key']); return CApiException.throw('tmpDemoBlocked'); else CApiKey.block(apiKey['key']); return CApiException.throw('blocked'); end; end; end; end; function run() local apiKey = ngx.ctx.REQUEST['key']; if not(apiKey) then return CApiException.throw('unauthorized'); end; apiKey = tostring(apiKey) -- check permanent and temporary banned checkBanned(apiKey); -- check api key apiKey = CApiKey.getData(apiKey); if not(apiKey) then return CApiException.throw('forbidden'); end; apiKey = JSON:decode(apiKey); if not(apiKey['is_active']) then return CApiException.throw('blocked'); end; apiKey['key'] = tostring(apiKey['key']); -- check is key in tmp blocked list if apiKey['mode'] == 'demo' then checkTemporaryBlocked(apiKey['key']); end; -- check requests count per second and per period checkRPS(apiKey); -- set apiKey's json to global parameter; in index.lua we send it through nginx to php application ngx.ctx.GLOBAL['api_key'] = JSON:encode(apiKey); end; 

Validator.lua

 module(..., package.seeall); local function checkApiVersion() local apiVersion = ''; if not (ngx.ctx.REQUEST['version']) then local nginx_request = tostring(ngx.var.uri); local version = nginx_request:sub(2,4); if tonumber(version:sub(1,1)) and tonumber(version:sub(3,3)) then apiVersion = version; else return CApiException.throw('versionIsRequired'); end; else apiVersion = ngx.ctx.REQUEST['version']; end; local isSupported = false; for i, version in pairs(config.app_params['supported_api_version']) do if apiVersion == version then isSupported = true; end; end; if not (isSupported) then CApiException.throw('unsupportedVersion'); end; ngx.ctx.GLOBAL['api_version'] = apiVersion; end; local function checkKey() if not (ngx.ctx.REQUEST['key']) then CApiException.throw('unauthorized'); end; end; function run() checkApiVersion(); checkKey(); end; 

Apikey.lua

 module ( ..., package.seeall ) function init() if not(ngx.ctx.GLOBAL['CApiKey']) then ngx.ctx.GLOBAL['CApiKey'] = {}; end end; function flush() CStorage.apiKey:flush_all(); CStorage.apiKey:flush_expired(); end; function load() local dbError = nil; local dbData = ngx.location.capture('/postgres_get_keys'); dbData = dbData.body; dbData, dbError = rdsParser.parse(dbData); if dbData ~= nil then local rows = dbData.resultset if rows then for i, row in ipairs(rows) do local cacheKeyData = {}; for col, val in pairs(row) do if val ~= rdsParser.null then cacheKeyData[col] = val; else cacheKeyData[col] = nil; end end CStorage.apiKey:set(tostring(cacheKeyData['key']),JSON:encode(cacheKeyData)); end; end; end; end; function checkNotEmpty() if not(ngx.ctx.GLOBAL['CApiKey']['loaded']) then local cnt = CHelper.tablelength(CStorage.apiKey:get_keys(1)); if cnt == 0 then load(); end; ngx.ctx.GLOBAL['CApiKey']['loaded'] = 1; end; end; function getData(key) checkNotEmpty(); return CStorage.apiKey:get(key); end; function getStatus(key) key = getData(key); local result = ''; if key ~= nil then key = JSON:decode(key); if key['is_active'] ~= nil and key['is_active'] == true then result = 'allowed'; else result = 'blocked'; end; else result = 'forbidden'; end; return result; end; function blockTemporary(apiKey) apiKey = tostring(apiKey); local isset = getData(apiKey); if isset then CStorage.tmpBlockedDemoKeys:set(apiKey, 1, config.app_params['ban_params']['time_demo_apikey_block_tmp']); end; end; function block(apiKey) apiKey = tostring(apiKey); local keyData = getData(apiKey); if keyData then ngx.location.capture('/redis_get', { ['vars'] = { ['key'] = apiKey } }); keyData['is_active'] = false; CStorage.apiKey:set(apiKey,JSON:encode(cacheKeyData)); end; end; 

Storages.lua

 module ( ..., package.seeall ) apiKey = ngx.shared.apiKey; RPS = ngx.shared.RPS; RPP = ngx.shared.RPP; banPermanent = ngx.shared.banPermanent; banTmp = ngx.shared.banTmp; tmpBlockedDemoKeys = ngx.shared.tmpBlockedDemoKeys; 

3. Additional services, such as inter-component communication using the AMQP protocol. An example is here .

4. As I wrote . Self-diagnostics module of the application with the ability to "intelligently" manage routes for passing the request through the backend. Still in development.

5. Adapters for APIs. In some cases, it is necessary to correct, supplement or expand existing methods. To not rewrite everything, LUA will help . For example, json <-> xml conversion on the fly.

6 ... many more ideas.

Benchmarks alone will not. Products are too complicated and RPS after Bench is highly dependent on many factors. However, for our products, we achieved a 20-fold increase in performance for the affected functionality, and in some cases everything became faster up to ~ 200 times.

Advantages and disadvantages

Tangible benefits. Everything that used to be 5 megabytes of code on php turns into a 100kb file on lua.
- development speed,
- the speed of the application,
- reliability
- asynchronous work with clients and backends, without breaking the nginx event loop,
- LUA sugar feel good! Korutin, shared dictionary for all nginx forks, sabrekvesta, a lot of binding.

Imperceptible cons.
- it is necessary to do everything carefully and remember about asynchrony and nginx event loop.
- the frontend works so fast that it may not like the backend. There is a direct connection between them, without interlayers. For example, I am sure that 10,000 queries per second LUA on the frontend will be easily chewed. But if at the same time it wants to go to the base, there may be problems.
- it is quite difficult to debug if something goes wrong.

By the way, while this article is being written, right at this moment our programmer talks about all this in detail on highload.

We are pleased to answer questions in the comments.

Finally, here you can find a small selection of information on the topic.