AMQP. Sending messages to RabbitMQ directly from Nginx (embedded Perl)

We use AMQP in our products. This is convenient, it allows you to scale better and allows you to add new modules and extensions to a complex system with virtually no problems. RabbitMQ is used as a broker. Nginx is used as frontend. Until recently, we everywhere used a bunch of php-amqp and librabbitmq-0.0.1 to work with a broker. But in some parts of the system it seemed to us redundant.

Task

Learn to send messages to the RabbitMQ broker directly from the front-end, bypassing php. Asynchronously as possible. Minimize programming and crushing of crutches.
Immediately, I’ll make a reservation that we wanted to optimize the entire service part of the system, which generates certain statistics, transmits the data further, and does nothing else. There is no content here and nothing needs to be given to the user. There are also no slow backends and nothing is blocked anywhere.

Ways to solve.

For Nginx written a considerable number of modules . We are already using or planning to use modules from the openresty assembly.
But there is almost nothing ready for working with amqp.
The Internet is also almost empty, this topic was popular in 2010, but this year it also stalled.
There is a Nginx module for 0MQ and a plugin for RabbitMQ, which is built but not working. And again 2010.
')
- You can write your own module for Nginx.
- you can use lua, but there are no bindings ready, you have to write.
- You can use something else, for example thrift and sharpen it for such a task.
But this is all a long time and we are still not programmers)))

A wrapper for librabbitmq on perl was found. Fortunately, nginx can embedded perl.

We pack everything in deb, there were no problems with building this particular module for nginx = 1.2.4.
You just need to make sure that the key is used - with-http_perl_module. And that the files of the assembled module are searched in the paths. We, for example, turned out like this:
objs/src/http/modules/perl/blib/lib/nginx.pm usr/lib/perl/5.14
objs/src/http/modules/perl/blib/arch/auto/nginx/nginx.so usr/lib/perl/5.14/auto/nginx


Net :: RabbitMQ we took from here .

A piece of nginx.conf

 http { perl_modules /etc/nginx/perl; perl_require rmqsux.pm; ... 

A piece of testsite.conf

 server { listen 80; server_name test01.dev; location / { perl rmqsux::amqppull; } } 

Sample script:

 whoareyou@test01:/etc/nginx/perl# cat rmqsux.pm package rmqsux; use nginx; use Net::RabbitMQ; use vars; rmqsux::connectmyrabbit; sub try (&@) { my($try,$catch) = @_; eval { &$try }; if ($@) { local $_ = $@; &$catch; } } sub catch (&) { $_[0] } sub connectmyrabbit { try { $mq = Net::RabbitMQ->destroy(); } catch { $mq = Net::RabbitMQ->new(); }; $mq->connect("192.168.1.100", { user => "test", password => "test", vhost => "testhost" }); $mq->channel_open(1); } sub disconnectmyrabbit { $mq->disconnect(); my $r = shift; $r->send_http_header("text/html"); $r->print("disconnected\n<br/>"); $r->status(200); } sub amqppull { my $r = shift; try { $mq->publish(1, "test", "qawsed"); } catch { rmqsux::connectmyrabbit; $mq->publish(1, "test", "oops died here"); }; $r->send_http_header("text/html"); return OK if $r->header_only; $r->print("send completed\n<br/>"); $r->status(200); $r->flush; } 1; __END__ 

Brought a synthetic test . You can bang more, but rabbitmq must be specifically tyunit.

 silenkov@sn00p:/home$ ab -c 300 -n 100000 http://test01.dev/ This is ApacheBench, Version 2.3 <$Revision: 655654 $> Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/ Licensed to The Apache Software Foundation, http://www.apache.org/ Benchmarking test01.dev (be patient) Completed 10000 requests Completed 20000 requests Completed 30000 requests Completed 40000 requests Completed 50000 requests Completed 60000 requests Completed 70000 requests Completed 80000 requests Completed 90000 requests Completed 100000 requests Finished 100000 requests Server Software: nginx Server Hostname: test01.dev Server Port: 80 Document Path: / Document Length: 20 bytes Concurrency Level: 300 Time taken for tests: 10.402 seconds Complete requests: 100000 Failed requests: 0 Write errors: 0 Total transferred: 13500000 bytes HTML transferred: 2000000 bytes Requests per second: 9613.40 [#/sec] (mean) Time per request: 31.206 [ms] (mean) Time per request: 0.104 [ms] (mean, across all concurrent requests) Transfer rate: 1267.39 [Kbytes/sec] received Connection Times (ms) min mean[+/-sd] median max Connect: 0 14 126.9 2 3007 Processing: 1 14 30.0 9 1543 Waiting: 1 13 30.0 8 1543 Total: 3 29 136.8 11 3230 Percentage of the requests served within a certain time (ms) 50% 11 66% 15 75% 16 80% 16 90% 20 95% 49 98% 56 99% 1009 100% 3230 (longest request) 

From the number of iterations, the results vary within 5%.

A bunch of php + librabbitmq gave us 300-600 rps, but here it is mainly in the final number of php workers.

Problems:
- connect is often broken under load, so such a tricky move in the script) We haven’t yet been able to determine the response threshold of this error. Up to 6000 rps nothing really breaks.
- it is not clear whether it will work asynchronously? You can rewrite everything to lua, then it will definitely be.
- the problems of this approach as a whole are not yet clear.
- need a containment mechanism for this car. Nginx is still faster than any backend. It will live at least one hundred thousand rps, but the backend is already gone. limit_req_zone here seems to be suitable, but it is necessary to select the burst under the load profile.

Performance in general, very happy. Messages are not lost, nothing has been flowing for the fourth day. Now all the tests will end, we will continue to do consume. There you have to parse everything and cram it into Postgres, Nginx also has this module and everything is fine with it.

Please refrain from commenting on bikes. We have a lot of rps and very few resources on such parts of the system, we have to invent and not that.

We will also be glad to any help and advice!