Lapis: Lua site in Nginx configs

Tl; dr Lapis (Lua) = RoR (Ruby) = Django (Python)

Introduction

Lua is a powerful and fast scripting language that can be easily integrated into C. Developed in PUC-Rio (Brazil).

LuaJIT

LuaJIT is the fastest implementation of Lua (JIT compiler), a true work of art. According to some estimates , it has a six-fold advantage over the standard Lua interpreter and in many tests beats V8. Developer Mike Pall (Germany).

LuaJIT can also bind C functions and structures on the Lua side (without writing bindings on C):

local ffi = require("ffi") ffi.cdef[[int printf(const char *fmt, ...);]] ffi.C.printf("Hello %s!\n", "wiki")

Nginx is one of the most efficient web servers developed by Igor Sysoev. Many large sites use Nginx. Starting from version 0.8, it is possible to directly embed the Lua language. Lua is executed in the Nginx process itself, and not taken out in a separate process, as is the case with other languages. The Lua code in the Nginx context runs in non-blocking mode, including database requests and external HTTP requests generated by the web application (for example, a request to the API of another site).

OpenResty is an assembly of Nginx with a lot of third-party modules, including non-blocking access to popular databases. Latest versions use LuaJIT to run Lua. Developer Yichun Zhang (USA, place of work: CloudFlare, the main developer of lua-nginx-module).

~~Sailor~~ MoonScript is a scripting language that is translated to Lua. Adds syntactic sugar, makes it easier to write certain things, such as list expressions; implements classes with inheritance. You could say that MoonScript for Lua is CoffeeScript for JavaScript. Developer Leaf Corcoran (USA).

lapis

Lapis is a web framework for writing Lua and MoonScript web applications that lives inside OpenResty. Developer Leaf Corcoran (USA).

What is the advantage of Lua in Nginx?

Tl; dr. All high-level language features and efficient use of resources under heavy loads.

For the answer, let's go back to the distant past, when all sites were served by the Apache web server.
Apache

Delays contribute to the red nodes and edges of the graph. Yellow shaded components located on the same machine.

Apache allocated a separate thread of the operating system, which read the request, performed the processing and sent the result to the user. (Modern Apache can be taught not to do this.) It turns out how many active requests, so many OS threads, and they are expensive. In such a scheme, most of the stream lifetime is spent not on processing the request, but on data transmission over the network, limited by the Internet speed of the user.

Nginx

How to deal with it? We need to instruct the operating system to monitor the transfer of data so that our web server can work only when the network has completed the next task. This approach is called non-blocking I / O and is implemented in most modern operating systems. Nginx web server uses this feature, due to which it can serve tens of thousands of simultaneous requests using just one OS thread.

Nginx, PHP

Thus, we optimized data transfer between the browser and the web server, but there is another bottleneck on which OS threads are idle: work with the database and external resources (for example, HTTP-API of another site). It is important to understand that it’s not so much the inevitable delays of the database itself or the external API, but the fact that our web application is idle idle until it receives an answer from them.

The usual solution : already in the web application itself to generate threads that we have so successfully disposed of in the web server. Effective solution : make the web application and database communicate in a non-blocking way. The web application sends the request to the database and immediately proceeds to the next request from the user. The database counts, returns the result, and the web application, when free, returns to processing the request from the user that originated the database request. This approach is used, for example, in node.js:
node.js

The database and external APIs are still shaded in red, as they may introduce latency. The advantage of the approach is that the web application is not just waiting for them, but processes other requests at this time.

Wonderful! Now let's see how external HTTP requests are programmed in node.js:

 var request = require("request"); request.get("http://www.whatever.com/my.csv", function (error, response, body) { if (!error && response.statusCode == 200) { console.log("Got body: " + body); } });

Suppose we want to download a file from a URL and do something with it. The result has to be processed in the lambda function. Uncomfortable? Is this the inevitable asynchronous fee? Fortunately, this is not the case; Let's see the similar code in Lapis:

 local http = require("lapis.nginx.http") local body, status_code, headers = http.simple("http://www.whatever.com/my.csv") if status_code == 200 then print(body) end

It is convenient to write the code for Lapis as if it were synchronous, but behind the scenes it is executed completely asynchronously. This is possible thanks to the active use of coroutines ( coroutines , green threads , and in Lua terminology just threads ). All code that processes a request from a user is executed in a separate coroutine, and coroutines can stop and continue in certain places. In our example, such a place was inside the call to the http.simple function.

Why are coroutines more efficient than OS threads? Have we not dragged all the overhead into the app? In fact, the key difference between coroutines and OS threads is the freedom of the programmer, where exactly the coroutine falls asleep and wakes up. (In the case of OS threads, the decision is made by the OS.) Started a query to the database - put the coroutine that generated the query to sleep. The answer came from the database - wake up the coroutine and continue its execution. We do a lot of work at the same time and all in one OS thread!

Note. A similar mechanism is about to appear in node.js.

Note. I advise you to read a great article about coroutines in the context of C ++. At the end of the article turned out asynchronous code, written as synchronous, and all thanks to coroutines. It is a pity that in C ++ coroutines are more of a hack than a generally accepted technique.

In addition, Lapis is executed directly in Nginx, which eliminates the overhead of transferring information between Nginx and the web application. Of course, node.js can be used as the main web server, without Nginx, but then the opportunity to use different features of Nginx is lost.

lapis

On the other hand, not everyone decides to send Lua code directly into the main Nginx. In this case, we run a separate Nginx with Lua on behalf of a separate user with reduced rights, but basically Nginx is setting a proxy.

lapis2

The effectiveness of Lapis is confirmed in a 10-gigabit benchmark . Lapis occupies a leading place at the level of C ++ and Java.

Lapis

On April 1, 2014, the April Fools' article “LUA in nginx: inline php noodles” was published on Habré. The article considered a comic code that implements PHP-like templates on Lua. Lapis was mentioned in comments to the same article. I did not find any other references to Lapis on Habré, so I decided to write myself.

Writing Hello World is boring. Let's write a simple web proxy instead on Lapis.

Install OpenResty

Install perl 5.6.1+, libreadline, libpcre and libssl and make sure that the ldconfig command is available (its parent folder may be missing in the PATH).

 $ wget http://openresty.org/download/ngx_openresty-1.7.4.1.tar.gz $ tar xzvf ngx_openresty-1.7.4.1.tar.gz $ cd ngx_openresty-1.7.4.1/ $ ./configure $ make # make install

Install Lapis

First you need to install LuaRocks (available in the main distributions).

 # luarocks install lapis

Create a web application

Create the backbone of the site:

 $ lapis new --lua wrote nginx.conf wrote mime.types wrote app.lua

If we hadn't passed the --lua option, the backbone in MoonScript would have been created.

Now let's implement the logic of our application in app.lua: a form for entering a URL is displayed on the main page of the site. The form is sent to / geturl, where the page is loaded at the specified URL and content is transferred to the user's browser.

 local lapis = require("lapis") local app = lapis.Application() local http = require("lapis.nginx.http") app:get("/", function(self) return [[ <form method="POST" action="/geturl"> <input type="text" value="http://ip4.me/" name="url" /> <input type="submit" value="Get" /> </form> ]] end) app:post("/geturl", function(self) local url = self.req.params_post.url local body, status_code, headers = http.simple(url) return body end) return app

The home page simply gives out HTML code with a form. Double brackets are another notation for strings in Lua. The / geturl page receives a POST request from the form, extracts the URL entered by the user from the form from it, downloads the content at that URL using the http.simple function (the OS thread is not blocked, see above) and shows the result to the user.

For http.simple to work, you need to change nginx.conf:

  location / { set $_url ""; default_type text/html; content_by_lua ' require("lapis").serve("app") '; } location /proxy { internal; rewrite_by_lua ' local req = ngx.req for k,v in pairs(req.get_headers()) do if k ~= "content-length" then req.clear_header(k) end end if ngx.ctx.headers then for k,v in pairs(ngx.ctx.headers) do req.set_header(k, v) end end '; resolver 8.8.8.8; proxy_http_version 1.1; proxy_pass $_url; }

This code creates in Nginx location / proxy, through which Lua makes external requests. In the main location you need to add set $ _url ""; Read more about this in the documentation .

Run our web proxy:

 $ lapis server

Click on the "Get" button. The site ip4.me shows the IP address of the server on which Lapis is running.

web-proxy result

If there is no path in the URL, then / proxy is used as the path. Apparently, this is a Lapis bug, for which a report has already been made.

Conclusion

In Lapis, Lua and Nginx there are many other interesting things, for example, asynchronous work with Postgres database, wrapper classes for database objects, HTML generation , powerful template language etlua , caching of variables between different Nginx workflows, protection against CSRF , two tools for testing and interactive Lua-console right in the browser. If the article finds a reader, I will continue the story about Lapis in other articles.

Without a doubt, Lapis has long outgrown the level of April Fool's joke and is rapidly gaining ground in the community of web developers. I wish you a pleasant study of these promising technologies!