How to make the site fly and save dozens of hours of system administration

The speed of your site, its stability and fault tolerance always depend on three components:

1. Platform (CMS) and its settings that affect performance (caching parameters, etc.)
2. Server configuration (real physical or virtual) and system software settings (web server, database, etc.)
3. Quality of development, code, integration with the platform.

Often, a web developer can write good quality code, but at the same time, little is understood in system administration and server configuration. A good sysadmin is rarely concurrently also a cool programmer.
')
In general, this is perfectly normal; everyone should do their work. But, unfortunately, in small web studios, which are the majority, there are rarely admins in the state. Hosting settings receive little attention. At best, they rely on the hosting support and default settings.

As a result, the site may be “limp” due to problems and “narrow” places in any of the components: CMS, hosting, development. The client does not go into the details and is not satisfied with the project as a whole. His negative is transferred to all: “Brake hosting! Awful system! Developers can't do anything! ”

Such a picture, of course, never suited us. And we decided to do something ...

The capabilities of the 1C-Bitrix platform itself, relating to performance, are truly enormous (this is a performance monitor , debugging tools, several caching options, support for web-based cluster systems , and much more) and require a separate article. We have already written about many things, we will tell you about something in the near future.

Today's post is about hosting, servers, VPS and their administration.

The simplest and, seemingly, the most obvious solution is to write a good detailed guide on configuring web systems and give it to everyone. And all will be well.

And we started just with this - we wrote detailed documentation (it was, it seems, 6 or 7 years ago). Then, based on it, several free training courses were released, which were constantly improved and adapted. As a result, they are now summarized in a general "Course for hosters . "

Despite the name and the fact that it was released by 1C-Bitrix, the course is relevant and extremely useful for all hosters in general and for administrators serving a particular fleet of web servers. After reviewing it and - all the more - having answered the control questions at the end, one can hope that there will be no more serious mistakes in configuring the web server.

But, unfortunately, even the presence of the most complete documentation and practical recommendations for setting up servers does not solve the performance problem for most projects! Good - right! - configurations are still very small.

There are many reasons for this ...

• Many administrators simply do not have time to fine-tune the system. Set, basic configuration performed - already good ...
• Low qualification of technical specialists - even in order to do everything “according to the manual”, we need some technical background and understanding of the processes taking place.
• “Historical heritage” - the use of configurations of 3-5-… -years old, at best - software upgrade. Strict adherence to the principle of "work - do not touch." Which, as it turns out, is not always applicable.
• Template configurations for all projects.
• Etc.

As a result, as our practice shows, there are many typical problems that almost always reduce the speed of the site.

I will step back a bit from the initial topic of the post and try to describe the most typical problems that we have encountered over the years. I hope that even if somewhere you recognize your own web server, a little understanding of the essence will help you ...

PHP as CGI

Fortunately, this PHP mode of operation is already becoming rare. Nevertheless, about a year ago there were quite a few such configurations.

What's wrong with CGI? That on each appeal to a script separate process PHP is started. It is long and resource intensive.

open_basedir

This parameter in PHP is responsible for restricting the access of scripts to certain directories. Very useful for configurations in which sites of different clients can work on the same server. The good goal is security, but the solution is implemented, to put it mildly, “not very” ...

First, there are quite a few options to “get around” the limitations set in open_basedir .

Secondly, setting this parameter (even to an “empty” value) has a very negative effect on the speed of PHP (file operations, for example, include). On an empty unloaded server, the speed of page generation can be reduced by 20-30%, and with a high load - by 2-3 times.

There are many alternatives to open_basedir. Starting with a separate copy of the web server for each client, ending with the use of chroot.

PHP precompiler not installed

APC , eAccelerator , XCache ... I hope these words are familiar to you.

The precompiler is used to optimize and speed up the execution of PHP scripts (precompiles the interpreted code, caches the result and then executes the already precompiled code).

The difference in performance on different projects can reach several times.

Not enough precompiler memory

The precompiler can be installed, but, for example, all settings are left “by default”. What could this mean? For example, by allocating too little memory for cached scripts, accessing new scripts will crowd out old data, and all the work of the precompiler will be inefficient.

Missing FrontEnd (nginx)

The two-tier architecture “Backend - Frontend” is a practically obligatory requirement for the stable operation of any more or less loaded web project.

This scheme of work solves several problems:

• all static is quickly given away by “light” (with minimal memory consumption) processes;
• backend is busy processing only "heavy" requests to scripts;
• on slow channels on clients, backend is not busy delivering content to the end client - it gives it to the frontend and is free to process subsequent requests.

The value of MaxClients in Apache is not adjusted.

On non-threaded servers, this parameter is responsible for the maximum number of processes that can be run for parallel processing of client requests.

Many people think that the more - the better. In many configurations, you can see the values ​​of both 50, and 150, and 256.

What does this mean in practice? Suppose a single Apache process can consume 40 MB of RAM. If MaxClients is set to 150, then at peak load (DDoS, habraeffect, errors in development, etc.), all processes require approximately 6 GB of RAM. Only under Apache.

If this amount of memory is not available, a swap will be used. And the general degradation of the whole system will begin. And even those requests that could be processed quickly will be processed for a very long time.

It is much better to limit MaxClients to a reasonable value. If the number of requests is greater, they will simply “queue up” and will be processed when busy processes become free. The system will be stable.

* * *

The list of such typical mistakes can be continued for a very long time ...

In our practice we came across them very often.

As a result, in 2009 we performed in a role that was quite atypical for us - not developers, but system administrators. And released a free product "1C-Bitrix: Virtual Machine . "

What it is?

VMBitrix (as we briefly call a virtual machine) is a ready-made VMWare Virtual Appliance image that can be run on a wide variety of VMware products (VMWare Server, VMWare ESX and ESXi, VMWare Workstation, VMWare Player, VMWare Fusion).

To be implemented in this image?

• Of course, the operating system has already been installed (the first images were on Ubuntu, later on - CentOS).
• Configured two-tier web server configuration (Apache + Nginx).
• Zend Optimizer + precompiler included in Zend Server CE package is installed. According to the results of load tests, he showed the best performance.
• MySQL is installed and configured.

Initially, we did not plan to widely distribute this image. The very first versions were not even public, but were used only for internal purposes.

VMbitrix essentially served as a reference environment to which you could navigate, deploying your own system.

Nevertheless, “1C-Bitrix: Virtual Machine” proved to be very popular. Hosters who could provide tariffs on the basis of VMware, began to use our image for new tariffs focused on placing projects on the 1C-Bitrix platform.

Gradually, requests began to appear "and make the same image under Hyper-V", "and make a template for Virtuozzo" ...

Gradually, we began to support almost all common virtualization environments.

Support for all current versions eventually began to require quite noticeable efforts, and as a result, about a year later, another free product appeared - “1C-Bitrix: Web Environment” (Linux) .

In essence, this is an RPM package that can be automatically deployed on CentOS (5, 6), Fedora (12-16), Red Hat Enterprise Linux (5, 6). Both 32-bit and 64-bit architecture are supported.

Thus, it does not matter at all whether it is a real physical server, or you rented a virtual machine (from Amazon , Scalaxy , Clodo.Ru or someone else), or bought a regular VPS .

On the "bare" system, just run:

# wget repos.1c-bitrix.ru/yum/bitrix-env.sh
# chmod +x bitrix-env.sh
# ./bitrix-env.sh

... and answer “yes” several times during the installation process. :)

(By the way, modern versions of the virtual machine, which continues to be supported, are assembled in this way - from the 1C-Bitrix: Web Environment package).

Actually, everything! After such a simple installation, you get a "Web environment" - now version 3.1.

How the product has evolved and what it can do now (it can do it right away, out of the box). I will describe some of the most interesting, in my opinion, "chips."

Convenient installer of any products "1C-Bitrix"

After installing the web environment, when you log into the server via HTTP, you are greeted by a very simple wizard that allows you to deploy an existing website from a backup copy or install a new one using any product, be it Site Management, Corporate Portal or this or other print solution.





Master is available in three languages ​​- Russian, English and German.

At the same time, the connection to the MySQL database is fully configured (in the /home/bitrix/www/bitrix/php_interface/dbconn.php file), so installation of any product follows the “shortcut” procedure: the system parameters are not checked (we know in advance that it is meets the technical requirements), the database connection parameters are not requested (everything is already configured).

Startup Menu

You will see this menu if you log into your machine via SSH:



Of course, it looks somewhat ascetic in comparison with modern beautiful web interfaces. :) Probably, someday we will make them for the web environment and the virtual machine - and we will end up with a full-fledged hosting control panel à la ISPmanager, cPanel or Plesk. :)

But even in this version it is a very powerful and convenient tool.

Web cluster

For about a year, the Web Cluster module has been actively developed in the 1C-Bitrix platform, which allows you to deploy any project not on one, but on several servers, thereby ensuring its scaling with increasing load and fault tolerance when one or several servers.

We have a detailed guide to setting up a web cluster , everyone can get acquainted with it.

But in order to save your time, you can use the wizard, which is part of the web environment:

• it will configure and start MySQL replication;
• configures nginx as a balancer for several nodes;
• configure a memcached server pool for a distributed data cache;
• will configure and start synchronization of these web servers using the csync2 utility.

The configuration process is approximately as follows:

• Start a virtual machine or deploy a web environment.
• Install through the wizard in the browser "1C-Bitrix: Site Management" one of the two senior editions (which includes the module "web cluster").
• Run the master node setting via the menu.
• We start one more virtual machine (or we expand a web environment on the second server).
• Through the menu of the first machine, add the slave node.

Further, all settings will be made automatically.

At a recent partner conference, Denis Sharomov, head of technical support, demonstrated the entire process of deploying a small test cluster in just 10-15 minutes.

In the same place, he demonstrated another great opportunity - switching any slave to master mode in the event of an accident at the master node.

You have a great opportunity to watch it all practically “live” - video recordings of the reports are posted on the conference website, including the report “Virtual Machine 3.0: How to launch a web cluster in 15 minutes with practical examples”.

Corporate Portal

“1C-Bitrix: Web Environment” is maximally adapted for any projects developed on the 1C-Bitrix platform (at the same time, any sites that use PHP and MySQL will work great on it).

But at the same time there are a number of individual "chips", specifically designed to work with the "Corporate Portal" .

For example, catdoc and xpdf utilities are automatically installed and configured in the product, which allow indexing and rendering documents of the most popular formats (MS Word, Excel, PowerPoint, Adobe Acrobat) on the fly on the portal.

The web server is immediately configured to work both over HTTP and over https. You can simply replace the self signed certificate with your purchased one - and work in a protected environment.

Further ... Immediately "out of the box" WebDAV is supported for connecting network drives.

Separately, we can mention the NTLM authorization setup wizard:



If the company uses Active Directory, and AD users are allowed to authenticate on the portal, then in addition to the actual settings of the portal it often takes a lot of “dancing with a tambourine” around the Apache and Nginx config files for NTLM authentication to work correctly.

Now it is not required, the whole process is automated!

(The entire setup process is described in detail in the blog of Nikolai Ryzhonin .)

Automatic configuration based on available resources

The web environment has a tricky /etc/init.d/bvat script.

Starting at system startup, it determines the amount of available resources (RAM) and, depending on them, automatically optimally adjusts:

• MySQL (buffer sizes, number of connections, temporary tables, etc.);
• MaxClients and related parameters in Apache;
• amount of available memory for Zend Optimizer +.

If after installing a virtual machine or a web environment you want to perform fine-tuning specifically for your tasks, disable this script so that it does not work every time the server is rebooted.

* * *

We somehow tried to calculate, and how much time it would take in “man-hours” :) to fully configure all software, deploy a cluster, configure NTLM authorization ... It turned out at least 200 hours! :)

Of course, this is the top estimate. However, even if you, for example, need to configure a VPS for stable operation of a regular site, it will take 1-2 days to install, configure and test everything.

With the “Web environment” or “Virtual Machine” “1C-Bitrix” this time can be saved and spent on something else. :)

PS We are often asked to lay out a detailed description of the web environment: with examples of scripts, configs, etc.

Such a “description” is the web environment itself . It is absolutely free. Put it on any machine or download our virtual machine . Log in on ssh - and study! All configs, all scripts - everything is completely open and accessible. :)