High-Load cluster organization with several nodes

In response to this, this topic decided to write its own ideas on how to build a cluster for highly loaded projects.

The topic is very good. Very nice to read this from the point of view of the developer.

By the nature of the activity I am engaged in the provision of hosting services. 90% of my work is setting up and administering servers for heavy stores, written on the magento engine. Therefore, I pay a lot of attention to performance.
')
During my work, I developed a certain vision of the ideal cluster for heavyweight projects. It:

- 2 WEB servers

- 1 DB server

- 1 load balancer.

Not new at all, but considerations are connected precisely with the load balancer.

How many who would not spit acid in the direction of the Apache, but I did not find a worthy replacement for him. Each product that had a chance to try had its flaws. There were all sorts of problems that I tried to solve with all my might, but in the end I sent all installations, such as lighthttp or nginx + php-fcgi, to the dump and returned the Apache with a nice set of modules.

Actually, it will run the php application on two web servers. In the same place, to taste, we include the necessary modules and we disconnect unnecessary.

Web servers:

Further pure abstract controversy on setting up web servers. After launching them into life and monitoring the load, you will have to change and reconfigure everything, but in the meantime you need to start somewhere.

Change the default mod_prefork settings. We set the ServerLimit and MaxClients directives to 100. Set the StartServers and MinSpareServers directives to the same values, for example, 50. We can ignore MaxSpareServers (commenting). Most likely, our server will not be able to process so many requests, and maybe it will be able to process more, it all depends on the application. That is why after starting the cluster, these values ​​will have to be changed. MaxRequestsPerChild can be set to 2000-3000.

Turn on keepallive and set it to a very small value. 3-ki will be enough. This is necessary in order to ensure that the content issued by the web server will be processed within a single get request.

Add the string “ServerTokens Prod” to apache.conf or httpd.conf to hide the version of our web server. It is necessary just in case. Our load balancer will hide this information from outsiders. But about it further.

Only the code will be stored on these servers. Static content (all sorts of images, fonts, icons), we put on the load balancer. To synchronize content, I would not recommend looking towards DRBD, since in the Primary-Primary mode it is set up only by young naturalists, and then in medicine labs. It is better to use rsync to synchronize content. A small crown task or a small script that will run every 2-3 minutes will do the trick. If the application writes some files (for example, logs) to its directories, then these files should be excluded from synchronization.

In terms of the “hardware” of the server itself, there is also plenty of room for thought to fly. Therefore, I’ll focus only on partitioning the hard disk:

Better to do a few sections. Separately boot, / var / log, / (root), SWAP and / var / www. It’s better if they (except / boot) are in lvm. This will allow you to easily add space, if necessary.

Breakdown at your discretion. For example: boot = 0.2Gb, / var / log = 5Gb, / = 5Gb, swap = 1Gb and / var / www = as much as your application needs.

I recommend to keep the application logs in / var / log. If the application does not support setting the location of the logs, then you can make custom folders, and configure bind-mounts in the right place.

fstab example:

/var/log/app_logs /var/www/app_html/log none rw,bind 0 0 

DataBase server:

Our database will live here. I will not tell much. Normal yourself DB server. Depending on what software you use, and customize. In terms of screw splitting, this is the same as for Web Servers, only instead of / var / www, for example, you will need to mount the partition in / var / lib / mysql, in case your databases rotate on mysql. Actually the same mounting logic for other engines.

The advantage of a separate DB server is that it will not load the server on which php is running. And vice versa. When working, two Web servers will use the same database.

Again, using lvm will allow you to easily add places on the fly to the desired section, if necessary.

Load Balancer:

The most delicious part. Do not underestimate this segment of the cluster, since it is the most important in my scheme. This is where all the magic will happen.

First you need to decide on the drive, as all static content will be stored here. The logic is the same - a separate section for logs, SWAPA and for the root of the file system. For media content is also a separate section. All in lvm'e.

We also put memcached here, and set up applications to connect to this server.

Next is to determine the load balancer. I would look in the direction of nginxa, since it supports ssl. But you can hang in front of him haproxy. Then 2 packet flow patterns will be obtained:

1. Client -> web_varnish: 80 -> haproxy: 80 -> web_backend

Client -> static_varnish: 80 -> nginx

2. Client -> haproxy: 443 -> nginx -> upstream_backend

The logic is that https traffic passes through haproxy and goes to a separate web-host nginxa, which describes the backend of the server. This allows you to do ssl handshake at the level of nginxa. Nginx gives statics, and nginx requests dynamic content from the backend servers described via the upstream module.

Http content will be cached using varnish, depending on the settings of the latter. In the settings of the varnish content is resolved according to the logic:

if you have requested something from css, js, ico, gif, jpeg, jpg, png, eot, ttf, swf, woff {

use nginx backend}

if something else {

use haproxy backend}

The scheme is not so simple. Why do I want to use haproxy? Just a fool? Not. It is haproxy that will allow us to control the number of connections that we will allow on our server backend. But all in order.

First set up varnish. Editing the warnish configuration file: in debian / ubuntu, this is / etc / default / varnish; in redhat / centos, this is / etc / sysconfig / varnish.

INSTANCE = $ (uname -n)

DAEMON_OPTS = "- a% external_ip%: 80 \

-T% external_ip%: 6082 \

-u varnish -g varnish \

-p thread_pools = 1 \

-f /etc/varnish/config.vcl \

-s file, / var / lib / varnish / $ INSTANCE / varnish_storage.bin, 1G "

Further we will describe my considerations in /etc/varnish/config.vcl

 backend haproxy { .host = "127.0.0.1"; .port = "80";} backend nginx { .host = "127.0.0.1"; .port = "802";} sub vcl_recv { remove req.http.X-Forwarded-For; set req.http.X-Forwarded-For = client.ip; # Remove unneaded request cookies: if (req.http.Cookie) { set req.http.Cookie = regsub(req.http.Cookie, "^(.*)$", "; \1"); set req.http.Cookie = regsuball(req.http.Cookie, "; +", ";"); set req.http.Cookie = regsuball(req.http.Cookie, ";(SESS[a-z0-9]|xyz+)=", "; \1="); set req.http.Cookie = regsuball(req.http.Cookie, ";[^ ][^;]*", ""); set req.http.Cookie = regsuball(req.http.Cookie, "^[; ]+|[; ]+$", ""); if (req.http.Cookie == "") { unset req.http.Cookie; } else { return (pass); } } # Set backend for static content... if (req.url ~ "*\.(css|js|ico|gif|jpeg|jpg|png|eot|ttf|swf|woff)$") { unset req.http.Cookie; set req.backend = nginx; } # Use apache as backend for dinamic content. else{ set req.backend = haproxy;} } sub vcl_fetch { set obj.ttl = 10m; set beresp.http.X-Backend = req.backend; if (beresp.status == 200 || beresp.status == 301 || beresp.status == 302) { if (req.url ~ "*\.(png|jpg|jpeg|gif|css|js|swf|ico)$") { unset req.http.Https; unset req.http.Cookie; unset beresp.http.set-cookie; return (deliver); } if (beresp.http.Content-Type ~ "text/html" || beresp.http.Content-Type ~ "text/xml") { if (req.http.Cookie ~ "frontend=") { return (hit_for_pass);. } } } } 

Further we set up a scheme for http:

1. Nginx Http configuration:

 server { listen 127.0.0.1:802; location ~* \.(ico|gif|jpeg|jpg|png|eot|ttf|swf|woff)$ { root /var/www/media; expires 30d; } location ~* \.(css|js)$ { root /var/www/media; expires 7d; } } 

2. Nginx High load page:

 server { listen 127.0.0.1:4433; ssl on; ssl_protocols SSLv3 TLSv1; ssl_certificate /etc/nginx/SSL/cert.pem; ssl_certificate_key /etc/nginx/SSL/server.key; location / { root /var/www/hl_page/; } } server { listen 127.0.0.1:803; location / { root /var/www/hl_page/; } } 

It remains to configure haproxy (file /etc/haproxy/haproxy.cfg).

 listen http 127.0.0.1:80 mode http option httplog clf balance roundrobin option forwardfor option httpchk OPTIONS * HTTP/1.1\r\nHost:\ www server 192.168.2.2:80 cookie SC1 maxconn 100 maxqueue 10 check inter 8000 fall 3 server 192.168.2.3:80 cookie SC1 maxconn 100 maxqueue 10 check inter 8000 fall 3 server backup 127.0.0.1:802 check port 803; 

Haproxy will accept connections on the loopback adapter on port 80. Next will send connections to the web server, while transferring the client's ip address (not sure if it will reach the final destination). For each backend 100 connections will be started, if the total value is exceeded for two backends - a beautiful page will be displayed with a message that the cluster is currently overloaded. The content of the page is the work of the hands and fantasies of those who host the site on the cluster.

This is where another delicious lies. (Sorry, those who don't like me using this word). The appearance of information that highload page is displayed in haproxy logs will be a kind of indicator that attendance is growing and you need to think about how to increase the cluster power or twist the settings. That is why I propose to use haproxy. In order for it to write logs you need to do 2 things:

1. Insert the following into the global section of the haproxy.cfg file

log 127.0.0.1 -14 local2

2. If using rsyslog, create the file /etc/rsyslog.d/haproxy.conf with the following content:

 # Save HA-Proxy logs $ModLoad imudp $UDPServerRun 514 $UDPServerAddress 127.0.0.1 $FileCreateMode 0664 local2.* -/var/log/haproxy/haproxy_all.log &~ 

Do not forget to create a folder for logs and configure logrotate. Logs will be great.

An additional bun for our cluster may be, for example, cacti, which will draw beautiful color graphics about the performance of our cluster. 3-4 hosts cactus can monitor without special load on the server. You can expand it on the same balancer. Instructions to read on the off-line cacti. Put in 2 minutes. Requires snmpd. At the same time, it will be possible to track whether it is possible to increase the limits of incoming connections to our backends. Remember I said that you still have to turn the MaxClients values ​​in the “prefork_module” apache2 settings? That's exactly the moment. If we do not have a large load on the server in the district of that time when a high load page is displayed, then the backends do a great job with the load and the limits can be increased by increasing the value of “maxconn” in the haproxy configuration.

The next step is to configure https. First, let's configure haproxy:

 listen https %externall_ip%:443 mode tcp balance source option ssl-hello-chk server 127.0.0.1:443 maxconn 200 maxqueue 10 check port 443 inter 8000 fall 3 server backup 127.0.0.1:443 check port 443 

Nginx Https configuration

 upstream apache { server 192.168.2.2:443; # first backend server server 192.168.2.3:443; # second backend server } server { listen 127.0.0.1:443; ssl on; ssl_protocols SSLv3 TLSv1; ssl_certificate /etc/nginx/SSL/cert.pem; ssl_certificate_key /etc/nginx/SSL/server.key; location ~* \.(ico|gif|jpeg|jpg|png|eot|ttf|swf|woff)$ { root /var/www/media; expires 30d; } location ~* \.(css|js)$ { root /var/www/media; expires 7d; } location / { proxy_pass https://apache; } } 

Instead of conclusion

It seems everything described what he wanted. Thanks to those who mastered this fiction. Just a few words in conclusion.

The great advantage of this scheme is the ease in increasing resources. Not enough power? Not a question - we clone a web node, run rsync, register its ip address in the balancer settings and voila - our cluster has become more powerful. Not enough space somewhere? Also not a problem - lvm our friend :)

A small problem can be migration, in terms of long-term planning.

I can not guarantee that the proposed configs will work for you. They are very rough. I tried to describe the idea itself.

If you want to have a very failover cluster, you can write another article.

If someone has enough wisdom to tune this up - contact me - I myself am wondering how it will work.