Tuning nginx

The article is based on my report on CodeCamp 2009 .

For many of us, the long-awaited day comes when the site’s audience begins to grow rapidly. Every morning, when we stop breathing, we look at the google analitycs charts and break into a smile when a milestone is taken in the next thousand visitors a day. As a rule, the growth of attendance does not coincide with the growth of the technical base and the site begins to slow down. Here the system administrator comes into play ...

Any project always has something to optimize: you can read optimization tips on webo.in , install eaccelerator, memcache, index search fields in the database. I assume that all this has already been done, and the site still slows down.
')
It's time to optimize nginx ...

Nginx compilation

When building, I usually follow the rule: “unplug everything that I don’t use”. So, rarely used modules that may not be useful to you: mail, mail_ssl_module, http_perl_module, http_flv_module, http_dav_module . If in the future some modules will be required, then it will take several minutes to recompile.

Modules that you want to include when compiling: http_gzip_static_module, http_stub_status_module .

Here's what a part of my spec file looks like to compile nginx (% nginx_datadir, ... spec file variables):

./configure \ --prefix=%nginx_datadir \ --conf-path=%nginx_etc/nginx.conf \ --sbin-path=%{_sbindir}/%{name} \ --error-log-path=%nginx_log/nginx.error.log \ --http-log-path=%nginx_log/nginx.log \ --http-client-body-temp-path=%nginx_spool/tmp/client \ --http-proxy-temp-path=%nginx_spool/tmp/proxy \ --http-fastcgi-temp-path=%nginx_spool/tmp/fastcgi \ --pid-path=%_var/run/nginx.pid \ --user=%nginx_user \ --group=%nginx_group \ --with-cc-opt="-I %_includedir/pcre/" \ --with-http_ssl_module \ --with-http_realip_module \ --with-http_addition_module \ --with-http_sub_module \ --with-http_dav_module \ --with-http_flv_module \ --with-http_gzip_static_module \ --with-http_stub_status_module \ --with-http_perl_module 

Nginx config - simple and clear

Nginx wrote admin for admins. This fact had a positive effect on the syntax of configs, as well as on the ease of configuration.
Sketch a simple config and analyze its directives:

 user nginx; #   ,      worker_processes 8; #     gettimeofday(),      timer_resolution 100ms; #       RLIMIT_NOFILE   . worker_rlimit_nofile 8192; #       -20  20 (     ). worker_priority -5; error_log /var/log/nginx/error.log; pid /var/run/nginx.pid; events { worker_connections 2048; # use kqueue;  freebsd (  ) use epoll; } http { include /etc/nginx/mime.types; default_type application/octet-stream; log_format main '$remote_addr - $remote_user [$time_local] $request ' '"$status" $body_bytes_sent "$http_referer" ' '"$http_user_agent" "$http_x_forwarded_for"'; access_log /var/log/nginx/access.log main; #  sendfile().  sendfile()   ,    , #     . sendfile on; keepalive_timeout 65; gzip on; gzip_min_length 1100; gzip_buffers 64 8k; gzip_comp_level 3; gzip_http_version 1.1; gzip_proxied any; gzip_types text/plain application/xml application/x-javascript text/css; # Load config files from the /etc/nginx/conf.vs directory include /etc/nginx/conf.vs/*.conf; } 

An example of the simplest configuration of a virtual server:

 server { listen 80; server_name _; location / { gzip_static on; root /var/nginx/html; index index.html index.htm; } error_page 404 /404.html; location = /404.html { root /var/nginx/html; } error_page 500 502 503 504 /50x.html; location = /50x.html { root /var/nginx/html; } } 

I commented on some directives, we will look at some later. The main thing to note is that the syntax is clear in most cases, even without documentation.

Nginx server monitoring

In the config nginx prescriptions section

  location = /stat { stub_status on; access_log off; allow xx.xx.xx.xx; deny all; } 

Now nginx statistics can be viewed at http://simple.com/stat

For convenience, it is also advisable to configure statistics for apache, which is hosted by nginx. To do this, we first install the mod_rpaf module, which allows apache to “see” the IP addresses of clients, and not the nginx IP ( here you can download and compile my version of rpaf srpm ), add to the apache config:

 LoadModule rpaf_module modules/mod_rpaf-2.0.so # # Mod rpaf # RPAFenable On RPAFsethostname On RPAFproxy_ips 127.0.0.1 xx.xx.xx.xx RPAFheader X-Forwarded-For 

There is an alternative module mod_extract_forwarded , which provides similar functionality (available in the Fedora repository). (Thanks Timosha )

We connect the statistics viewing module:

 ExtendedStatus On SetHandler server-status Deny from all Allow from xx.xx.xx.xx 

Now statistics for apache can be viewed at http://simple.com/apache-stat

Types of static content

I conditionally divide the statics into 2 categories:

“Easy” content : html, css, js, xml, rss, txt. It lends itself well to compression, requires little storage space. Using nginx in any case will give a noticeable performance boost.

“Heavy” content : photo, video, audio files. The bottleneck is, first of all, the disk system, the size of the RAM, the bandwidth of the channel. The task of distribution of this type of content is divided into 2 - the storage of content and, in fact, the distribution of content. With the help of nginx, you can achieve minimal memory and processor time, but as the load increases, you will still need to build a RAID array.

What to do when the site starts to slow down

Well, firstly do not panic. Perhaps at this very moment popularity comes to your site :)
As a rule, it is not nginx that is slowing down the backend (the server that generates dynamic content) or, as often happens, the database server, the disk space has run out, a super-duper analyzer of something has started and zagreb all the resources, ...
In this article we will consider the case when you suspect it is nginx. What can be tweaked to temporarily alleviate the suffering of the server?

1. Try to increase the number of worker_processes, the author of nginx advises to install them by the number of cores. I varied this number approximately in the range of “number of cores” - “number of cores x 2”, while achieving faster establishment of the connection, but not always more efficient processing of the http request. Do not forget that there is still worker_connections (the maximum number of horses for one worker)
2. Installing worker_priority to -5 or less (up to -20) will help. Be careful here, as other services may begin to slow down noticeably. On our servers, this parameter is set to -5. The smaller the value, the higher the priority for nginx.
3. With a large number of returns of small files and a slow hard drive, temporary shutdown of access_log off logs can help.
4. UPD: Mount the disk from which the distribution is with the noatime option, this will reduce the number of write operations to the disk. (Thanks for the idea of coolspot , pwlnw , alfa )
5. Look for a bottleneck, it can be eliminated. The following commands will help you: top, iostat, df -h, iptraf
6. Add RAM or upgrade your disk system (for example, install a RAID array, you can experiment with an SSD hard drive)

Optimization techniques

"Easy" content

1. Virtual disk
   Create a virtual disk (tmpfs or ramfs), js, css, images folders (if there is a small amount of images related to the design and not the content), we transfer it there and in the nginx config, separately we write

   location / js / {
   root /var/www/img_virtual/auto.ria.ua/js
   }
   ...
   

   In order to create a virtual disk automatically upon reboot, add in / etc / fstab

   none / var / www / img_virtual tmpfs size = 1g, mode = 1777 0 0

   (at the start of the system a 1G disk will be automatically created)
   Here you should pay attention to the following circumstance: if a static file gets into the system cache, then the speed of its return from the system cache is equal to the speed of return from the virtual disk. In other words, if the total amount of all distributed “light” statics is small, then there is no need for a virtual disk (Thanks to maxp for prompting me to test and see this)
   
2. Content compression gzip
   Run in our virtual folder

   for i in `find ./* -type f -name '* .js'`; do echo $ i; gzip -c -9 $ i> $ i.gz; done;
   for i in `find ./* -type f -name '* .css'`; do echo $ i; gzip -c -9 $ i> $ i.gz; done;
   

   add the line gzip_static on to the nginx config:

   location / js / {
   gzip_static on;
   root /var/www/img_virtual/auto.ria.ua/js
   }
   

   You can also enable online packaging for dynamic files:

   location / {
   gzip on;
   gzip_min_length 1100;
   gzip_buffers 16 8k;
   gzip_comp_level 3;
   gzip_types text / plain application / xml application / x-javascript text / css;
   
   root /var/www/auto.ria.ua/
   }
   

3. Headers for proxying content
   Specifying static content in the headers will also reduce the load. For this we will use the expires directive. If the content does not change, you can use expires max. Even expires 1d will give good results.
4. File Descriptor Caching
   This will give a performance boost if you have a lot of small files with an expanded directory hierarchy. You can also cache access to non-existent files. It will look something like this:

   location / {
   root / var / www /;
   
   
   open_file_cache max = 1024 inactive = 600s;
   open_file_cache_valid 2000s;
   open_file_cache_min_uses 1;
   open_file_cache_errors on;
   }
   

"Heavy" content

1. First, you need to pay attention to whether the swap is not used in the delivery of content, if so, then this can be a problem if the swap is on a regular SATA hard drive. Swap loves the sendfile kernel metod, this is certainly a progressive technology, but using swap will significantly affect the return and then try sendfile to turn off with sendfile off and select the optimal output for output_buffers, for example output_buffers 2 64k ;
   Configuration example with sendfile disabled:

   sendfile off;
   tcp_nodelay on;
   output_buffers 2 64k;
   

   I draw your attention to the fact that much will depend on the kernel used by the system, this helped me on the kernels> = 2.6.27.
2. If RAM permits, create a virtual disk on which you place the most "requested" files. With time, say, every 10 minutes, you can correct the list of these files. Now we can apply the try_files directive:

   location / {
   root / var / www /;
   try_files / img_virtual / hot / $ uri storage ;
   }
   
   location storage {
   proxy_pass http: // backend;
   proxy_set_header Host $ host;
   }
   

   If the file is not found, the backend will be accessed on the virtual disk.
   
   As a rule, "hot content" is less than 10% of the total.
   
   If there is no possibility to write your cache generation program, use the proxy_store directive

   location storage {
   proxy_pass http: // backend;
   proxy_set_header Host $ host;
   
   proxy_store on;
   proxy_store_access user: rw group: rw all: r;
   proxy_temp_path / var / www / img_virtual / hot /;
   root / var / www / img_virtual / hot /;
   }
   

   With this configuration, each requested file is placed in the cache.
   Over time, the cache must somehow be cleaned, the easiest way to run on the crown:

   cd / var / www / img_virtual / hot /
   find ./-type f -amin +60 -delete
   

   (if the file from the cache is not requested for more than 60 minutes, we delete it)
3. If the storage is large, it takes terabytes - the operative question is not solved. You can build RAID on the frontend. I do not advise using fake-raid, it is a headache when upgrading and when light is lost! We take more SAS screws, we take a full-fledged RAID controller (no hostraid!). We mount swap, spool and cache there. SSD-hard drives can be used for infrequently changing content and infrequent cache rewriting. It works quickly, such hard drives do not have such characteristics as seek-to-seek, low power consumption (for example, for Intel X25-M 0.15W), good return speed (up to 250 MB / s).
4. Caching proxied requests.
   On March 23, 2009, the next beta of nginx 0.7.44 was released, in which experimental support for caching proxied requests appeared. It is difficult to overestimate the importance of this event for nginx users. The author nginx gives us a powerful tool in managing the distribution of a large amount of static.
   Why do you need such a cache? The answer to this question is mainly related to the different “cost” of a disk operation and a network operation. “Going to disk” in many cases is significantly “cheaper” than contacting the network. The main task of such caching comes down to the necessary minimum of network operations and “intellectual management” of disk cache.
   Learn more about setting cache of proxied requests.