Cache static

There is an opinion that nginx is a great tool for returning statics.
There are articles that describe the sendfile or aio settings for "improving" performance.
On Habré there is something to read about setting up proxy_store with proxy_cache to minimize problems with the brains of the site.
Even in QA, sometimes there are questions about caching pictures , for example.

Why engage in this nonsense! - Experienced users say - OS knows you better how to cache files! With a cache and a prefetch in modern OS, more precisely FS, there are no problems! Why produce your caches and lists of popular materials and all that? ...

There is only one harmful "but" - in the nginx runtime environment (generally Linux), the concept of " file " and in general "file system" is just a concept .
And once, when I, having mounted the server via sshfs, updated one script, the magic happened:
1. On each page there were 4 more pictures.
2. Servers are dead.
What to do - the pictures were stored on glusterFS. Come full fuse.

GlusterFS, like the FUSE interface, greatly eats up performance, but it's worth it .
Initially we had one server. Then two, the second of which mounted the www from the first to nfs. Then three, four.
Not very reliable, but historically so. To save the situation, we needed some kind of completely transparent system, in-place replacement of nfs. Cluster FS.
Now there is a raid 10 on all servers, and after a year and a half of operation, it fell a couple of times. Including absolutely. Gluster silently picked up a copy of the FS on a new piece of iron and everything continued to work. Glaster knows his job.

But in fact, the story began a little wrong.

The reason was different, but the problem solving would be the same.
There is nothing worse than the fact that " historically ."
And in one of my projects, "historically," the situation when the image gives php.
At one time, it was necessary to make a file downloader with various ACLs, resizes, and watermarks. That was almost 10 years ago. The mechanism took root and has not changed since.
Now there are many specialized solutions for solving such problems ( elliptics , backpack, and others; even better, CDN). But I do not want to spend man-months on the transition. I want to sleep, not work.
Yes, and php chtuka, in principle, not bad. But it has its limits.

But the problem of brake backends solved

Habr, once again prompted a solution - proxy_store and proxy_cache .
But I can't use proxy_store - I have nowhere to save, and there are also problems with proxy_cache.
There is one big such “but” - many years ago I was already confronted with the performance of uploading files and decided “let better nginx give the final files for me”.
X-Accel-Redirect.
Problem - nginx cannot cache a response from a proxy if this is not an answer, but a command. X-Accel-Redirect - not cached .
No exit?

The original problem was fixed

Now back to the top of the topic.
We need to give statics, this time without the intervention of "smart" backends. Just static, but with glusterfs.
There are simply no guidelines for caching statics. It is believed that the OS itself is not a fool. For static, there is only open_file_cache and setting up the file transfer itself (sendfile / aio and company).
What to do?
Let's complicate the problem, and try to solve the problem exclusively with nginx, without making any changes to other codes.

normal config

# Static files location location ~* ^.+\.(jpg|jpeg|gif|png|ico|css|js|swf|txt|ico)$ { root /var/www/kashey; } 

')

Patched config

Nginx makes proxy_pass on itself, caching the result. Perhaps here you can use fcgi, but so - more clearly.

 # Static files location location ~* ^.+\.(jpg|jpeg|gif|png|ico|css|js|swf|txt|ico)$ { set $dopass 1; proxy_set_header Host $host; #    "" -      if ($http_fiberpass) { root /var/www/kashey; # proxy_pass   set $dopass 0; break; #  break   } # ,    proxy_cache_valid proxy_cache static_cache; #  "" proxy_set_header fiberpass "nginx"; if ($dopass) { #    . proxy_pass http://127.0.0.1:80; } } 

And no magic.
In the case of X-Accel-Redirect, the scheme is as follows:

1. nginx get request
2. sends itself, unfortunately the request is complete and real.
3. forwards to proxy
4. receives the answer and gives the file
5. The answer is cached.
6. ???
7. PROFIT

In the case of regular statics, it is more difficult - root does not finish execution, break and return also do not work. Proxy settings cannot be inside a conditional block.

What does this give?

In sense with X-Accel-Redirect with apache the shaft of requests is removed. On a loaded system, the difference can be 50 times.
But this is understandable - php itself, and through Apache, all the more, not the most reactive solution.
The request latency is less than 40ms, I personally did not get it (total php + gluster time).
In the case of using the variant with proxy_cache, the speed of return is equivalent to the rate of return of static.

And what about the statics?

Return with gluster - 13ms is the normal time for a system without a cache.


Return from the cache - 1ms - this is wonderful.


It is important to note that the servers are located in Germany, and the customers in Russia.
Normal pings to the site 60-100ms.
So for the client, the acceleration is almost negligible.

But it is noticeable for the server.
1. The load on the servers has decreased, including in parrots Load average and soft-irq.
2. The traffic between the servers has decreased (it can be solved by turning on WORM (Write Once Read Many Volume), but this is no longer “transparent”).

3. Everything began to work a little faster.

As a result, the work that reduced the reaction rate at the client by 10% resulted in a 10-fold acceleration of static recoil. That reduced the total load on the server about 20 times.
The reverse statement, unfortunately, will also be true - no matter how hard you try, the client may not notice.

PS: Many familiar system administrators did not even allow the thought of the possibility of such a scheme. Does not allow to cache / proxy - it means it is impossible.

Good programmers are not always good system administrators, and system administrators are usually not programmers at all.
These heroes always go around, ford, and on bicycles.

PPS: by the way, I am not a sysadmin. And, perhaps, did everything wrong.