Comet – application for Mochiweb with a load of 1 000 000 users. Part 1/3

Part 2
Part 3

In this series of articles, I’ll share what I learned empirically about how Mochiweb handles a large number of open connections, and show you how to create a Comet application using Mochiweb, where each connection is registered in the router. We will end up with a working application that is able to cope with 1,000,000 parallel connections, and find out how much memory we need for this.

In part one:
• Creating a simple Comet - an application that sends a message to customers every 10 seconds.
• Configure the Linux kernel to support a large number of connections.
• Creating a testing utility to create a large number of connections.
• Determine the required amount of memory.
')
The following parts of this cycle will tell how to build a real system, show additional tricks to reduce memory usage, and contain tests with 100,000 and 1,000,000 concurrent connections.

It is assumed that you are familiar with bash, and a little with Erlang.

Creating a testing application

In short:

1. Install Mochiweb.
2. Run: /your-mochiweb-path/scripts/new_mochiweb.erl mochiconntest
3. cd mochiconntest and change src / mochiconntest_web.erl

This code (mochiconntest_web.erl) only accepts connections and sends a welcome message, and one message every 10 seconds to each client.

-module(mochiconntest_web). -export([start/1, stop/0, loop/2]). %% External API start(Options) -> {DocRoot, Options1} = get_option(docroot, Options), Loop = fun (Req) -> ?MODULE:loop(Req, DocRoot) end, %we'll set our maximum to 1 million connections. mochiweb_http:start([{max, 1000000}, {name, ?MODULE}, {loop, Loop} | Options1]). stop() -> mochiweb_http:stop(?MODULE). loop(Req, DocRoot) -> "/" ++ Path = Req:get(path), case Req:get(method) of Method when Method =:= 'GET'; Method =:= 'HEAD' -> case Path of "test/" ++ Id -> Response = Req:ok({"text/html; charset=utf-8", [{"Server","Mochiweb-Test"}], chunked}), Response:write_chunk("Mochiconntest welcomes you! Your Id: " ++ Id ++ "\n"), %% router:login(list_to_atom(Id), self()), feed(Response, Id, 1); _ -> Req:not_found() end; 'POST' -> case Path of _ -> Req:not_found() end; _ -> Req:respond({501, [], []}) end. feed(Response, Path, N) -> receive %{router_msg, Msg} -> % Html = io_lib:format("Recvd msg #~w: '~s' ", [N, Msg]), % Response:write_chunk(Html); after 10000 -> Msg = io_lib:format("Chunk ~w for id ~s\n", [N, Path]), Response:write_chunk(Msg) end, feed(Response, Path, N+1). %% Internal API get_option(Option, Options) -> {proplists:get_value(Option, Options), proplists:delete(Option, Options)}. 

Launch Mochiweb application

 make && ./start-dev.sh 

By default, mochiweb listens to port 8000. If you are working on a home computer, you can check with any web browser: http: // localhost: 8000 / test / foo .

Below is the command line test:

 $ lynx --source "http://localhost:8000/test/foo" Mochiconntest welcomes you! Your Id: foo<br/> Chunk 1 for id foo<br/> Chunk 2 for id foo<br/> Chunk 3 for id foo<br/> ^C 

Yes it works.

Configuring the Linux Kernel for a large number of tcp connections

Save yourself some time and configure the kernel before conducting tests with a large number of connections, or your test will not work, and you will see a lot of "Out of socket memory" messages.

Here is the sysctl setting:

 # General gigabit tuning: net.core.rmem_max = 16777216 net.core.wmem_max = 16777216 net.ipv4.tcp_rmem = 4096 87380 16777216 net.ipv4.tcp_wmem = 4096 65536 16777216 net.ipv4.tcp_syncookies = 1 # this gives the kernel more memory for tcp # which you need with many (100k+) open socket connections net.ipv4.tcp_mem = 50576 64768 98152 net.core.netdev_max_backlog = 2500 # I was also masquerading the port comet was on, you might not need this net.ipv4.netfilter.ip_conntrack_max = 1048576 

Put them in / etc / sysctl.conf, then run sysctl -p to apply them. There is no need to reboot the system; now the kernel should be able to handle many more open connections.

Creating a large number of connections

There are many ways to do this. Tsung is pretty sexy, and even less sexual ways to create httpd spam with more requests (ab, httperf, httpload, etc.) are enough. None of them is ideal for testing our application, so I wrote a basic Erlang test for this purpose.

Just because we can, does not mean that we should ... One process for each connection would definitely be expensive. I used one process to load a URL from a file, and another process to establish a connection and receive messages from all http connections (and one process as a timer to print a report every 10 seconds). All data received from the server is discarded, but the counter is incremented, so we can track how many HTTP messages were delivered.

 -module(floodtest). -export([start/2, timer/2, recv/1]). start(Filename, Wait) -> inets:start(), spawn(?MODULE, timer, [10000, self()]), This = self(), spawn(fun()-> loadurls(Filename, fun(U)-> This ! {loadurl, U} end, Wait) end), recv({0,0,0}). recv(Stats) -> {Active, Closed, Chunks} = Stats, receive {stats} -> io:format("Stats: ~w\n",[Stats]) after 0 -> noop end, receive {http,{_Ref,stream_start,_X}} -> recv({Active+1,Closed,Chunks}); {http,{_Ref,stream,_X}} -> recv({Active, Closed, Chunks+1}); {http,{_Ref,stream_end,_X}} -> recv({Active-1, Closed+1, Chunks}); {http,{_Ref,{error,Why}}} -> io:format("Closed: ~w\n",[Why]), recv({Active-1, Closed+1, Chunks}); {loadurl, Url} -> http:request(get, {Url, []}, [], [{sync, false}, {stream, self}, {version, 1.1}, {body_format, binary}]), recv(Stats) end. timer(T, Who) -> receive after T -> Who ! {stats} end, timer(T, Who). % Read lines from a file with a specified delay between lines: for_each_line_in_file(Name, Proc, Mode, Accum0) -> {ok, Device} = file:open(Name, Mode), for_each_line(Device, Proc, Accum0). for_each_line(Device, Proc, Accum) -> case io:get_line(Device, "") of eof -> file:close(Device), Accum; Line -> NewAccum = Proc(Line, Accum), for_each_line(Device, Proc, NewAccum) end. loadurls(Filename, Callback, Wait) -> for_each_line_in_file(Filename, fun(Line, List) -> Callback(string:strip(Line, right, $\n)), receive after Wait -> noop end, List end, [read], []). 

Every connection we create requires a port, and by default their number is limited to 1024. To avoid problems, we need to change the ulimit parameter for the shell. This is done in / etc / security / limits.conf, but requires a reboot of the system.

 $ sudo bash # ulimit -n 999999 

We can also increase the range of available ports to the maximum:

 # echo "1024 65535" > /proc/sys/net/ipv4/ip_local_port_range 

Generate a file with the URL for our testing utility:

 ( for i in `seq 1 10000`; do echo "http://localhost:8000/test/$i" ; done ) > /tmp/mochi-urls.txt 

Next, we compile and run our utility using the Erlang console:

 erl> c(floodtest). erl> floodtest:start("/tmp/mochi-urls.txt", 100). 

The code will establish 10 new connections per second (i.e., 1 connection every 100 ms).
Statistics will be displayed in the form {Active, Closed, Chunks}, where Active is the number of connections currently established, Closed is the number of connections that were terminated for some reason, and Chunks is the number of messages sent from Mochiweb. Closed must remain at 0, and Chunks must be larger than Active, because each active connection causes many messages (1 every 10 seconds).

The size of the Mochiweb process with 10,000 active connections was 450 MB - this is 45 KB for each connection. The CPU utilization on the machine was virtually zero.

Findings.

This was the first attempt. 45 KB for each connection seems to be quite large - you can probably collect something on C using libevent, which could do the same, at a cost of 4.5K for each connection (this is only an assumption, if anyone has a similar experience, please leave a comment). If we take into account the amount of code and the time it took to do this in Erlang compared to C, I think that the increased memory used is more excusable.

In future articles, I will show the message router (so we can uncomment lines 25 and 41-43 in mochiconntest_web.erl), and talk about some ways to reduce memory usage. I will also show test results with 100,000 and 1,000,000 connections.

Update: Part 2