New Year's gift to hosters: How to place on the server 10,000 clients or even more

Today I want to make a New Year's gift, and since the most valuable thing in the world is experience, knowledge and technology, I want to share it all with you. I hope that my three-year experience as a system engineer / architect will help some hosters to discover new horizons for themselves.

It should be noted that, under the client, a normal LAMP account (Linux + Apache + Mysql + PHP) is perceived, with one or two virtual hosts and the default php.ini working with all options. The main one is 16MB RAM and 30 seconds of execution. Quite enough for most engines - Wordpress, Drupal and Joomla.

Iron

Of course, the most important thing is to choose the server correctly. Experimentally, the optimal combination was found - Dell 905 + MD 1000 basket. A little more about the configuration. Of course, the server will work with two 6-nuclear Opteron. The memory for ~ 15k clients, namely, I worked so much on my server, 128G was enough, but you can try 64G + swap, if of course you don’t feel sorry for your clients;)
')
There is nothing to say about the basket - everyone needs a place for the site. The only thing - you need to competently break the stack, but more on that later. You can use SAS, but you can - cheaper SATA. In my case, SATA was enough for 15k, but if you plan more, you will have to take SAS.

Soft

Of course, only the web should work on the server. MySQL is only on a separate server. Mail is the same, and sending is something easy, like ssmtp, and not monsters like exim / sendmail. But in general, it is not important. There are more important things to consider.

First of all, this is the ratio of the use of RAM and bit depth. I am sure that everyone knows why 64-bit binaries take up more RAM when working. But who knows what the bottleneck is when working with many sites? The correct answer is their logs. It is the logs that take up most of the server's resources, since working with a write disk is the slowest part of the system. The conclusion is simple - you need to do something with the logs and use 32-bit software instead of 64-bit software.

With logs, at first there was an idea to make mod_log_spread under the second Apache. Or even use the first one only for logs. But a static survey helped - most clients did not use web server logs and simply turned them off by default. Those who requested - included on request. And for recording logs naturally used a separate disk.

Most delicious

But what about 32-bit software? And with the fact that with more than 1000 virtual hosts Apache behaves worse and worse? And here OpenVZ will help us! Yes, yes, that is how we break the server into 32-bit virtual machines, let us have our own Apache in each one - and voila, it all works fine, since the same virtual players are good at sharing memory with each other, thanks to OpenVZ. And given that you can connect a separate disk as an arbitrary directory for each virtual machine - the problem with the logs has already been solved only because of this!
Well, now a little about how it all worked from a technical point of view. I’ll say right away that I reassembled the Apache, throwing out a lot of unnecessary, including ssl, and choosing MPM Worker - the thread apache. Naturally, each virtual machine used all 12 cores, and the number of processes for Apache was also 12. Like the workers for Nginx.

Speaking of Nginx - did I say that there you need to give all the static directly, without touching the Apache? Probably not, since it goes without saying. But how to work PHP - it is worth talking in more detail.
Of course, in MPMWorker, mod_php cannot be run on production - whatever the homegrown "administrators" would say, it falls and will fall because many of its extensions do not know how to work in threads, which has already been discussed a lot. So the only option left is mod_fastcgi + PHP-FPM. And it is natural that in each virtualka twisted 12 PHP processes.

Thus, each virtual machine was a bundle of 32-bit applications: nginx + apache-worker + php-fpm. Raid-1 was created for each virtual machine in the basket, a separate shared disk was made for logs, which was connected to the logs directory. Then I looked at it all and thought - and if we have a common disk under the logs, why not make one common nginx for all virtuals? After all, OpenVZ works with virtual machines at the file system level, rather than block devices, and accordingly all files are accessible from the hardware node, you only need to substitute the path to the virtual directory.
No sooner said than done. Production server on 10k virtualok should consist of:

• 32-bit virtual machines with 1k hosts on each
• A dedicated disk for each virtual machine
• Apache in the thread mode + PHP-FPM, taking into account the processes by the number of cores
• Single nginx on all virtuals for the return of statics and acceleration
• Dedicated disk for recording logs + minimizing them

Well, in the end - a bonus about the filer. If you fill the rack with such iron + give 1 server more than the baskets, then a very fault-tolerant system comes out. All data lies in the basket, and on the server - a regular script that runs each found disk as an OpenVZ + container connects the dedicated disk to each virtual machine for logs. If one of the servers fails, then we simply connect the backup to the basket, and the broken one is quietly returned for repair.

PS The cost of iron for the placement of ~ 500k customers with discounts (purchased through the Data Center in the US) was ~ 90k $. Or in other words - the cost of half a hundred customers - just nine bucks without taking into account rack, food and traffic.