Probably, everyone already knows that one of the main bottlenecks of the servers is the disk subsystem. This is especially noticeable on web servers and large DBMSs. Hard drive manufacturers are in a constant race for performance, but you can't trample against physics - the hard drive head cannot dangle at the speed of light :).

SSD come, it would seem, here it is - happiness! No mechanics, no need to wait until the head reaches the desired point (especially if the data is fragmented or the OS tries to read a lot of things at once). But no - it's expensive, unreliable, there is little space - in general, you won't put it on the server.

What solution? That's right, combine! The task is to get the speed and access time of the SSD and the reliability and volume of the HDD. There are hardware solutions, but we are poor economically savvy - so we will do it programmatically, using flashcache.

')

What is flashcache

We use flashcache from facebook, which for some reason there are almost no mentions on Habré.

Flashcache was created inside Facebook to speed up MySQL (and other DBMS) on its servers and in 2010 was laid out as open source.

Its essence is simple: it is a transparent cache on the SSD. On the basis of two block devices (usually hdd - “from where” and ssd - “to”), another one is created, cached, which is used by the system. Transparent cache on SSD allows you to combine the advantages of both HDD and SSD.

How it all began

Somewhere about two years ago, a friend of mine seriously took up the project of the site with the selection and sale of cars from Germany. The site previously lived on a shared-hosting and terribly stupid - the engine was written incomprehensibly when and it is not clear by whom, the main page was generated in about 2-3 seconds. The rapid growth in attendance led to a move to a separate one, where he still lives - the promotional i7 from fastvps (hetzner reseller). After a while, load average in the region of 8-10 became almost the norm - it became clear that it was necessary to revise the code.

I laughed at one moment for quite a long time - there was an include on the main page, which worked for about 3 seconds and it was not clear what it was doing, but it does not work without it. The results of the revision, though predictable, are sad all the same: it turned out that it is easier to rewrite everything from scratch (which we are doing now) than to rule what we have. The rendering of some blocks in SSI with caching helped a bit (then nginx + php-fpm was already running there), but I had to do something. Writing it all over again - for a long time (and the owner is still not quite happy with this decision - hello, Uncle Sasha! :)), they began to look for alternative solutions. The edge of consciousness pushed to the surface somewhere heard the thought - you can cache on the SSD. The thought spun and sank.

Half a year ago, a turning point came - two or so ( hard-drive ) hard disks flew into the Marlborough Country at once in RAID-1. While hetzner was changing HDD, they decided to die, so with music! And ordered in addition also SSD.

What came of it

It was set up all six months ago, it was done immediately with flashcache, so to build graphs “before” and “after” I had to stop everything.

So, on the charts, in chunks from left to right:

• work with flashcache
• break off
• Work without flashcache directly from the HDD (poor zabbix because of the load did not have time to save the data - let's understand and forgive)
• a break to turn on everything back (here I made a mistake, about which later)
• work with flashcache
• break for error correction
• normal operation in normal mode (with flashcache)

CPU load

Back in the good old days of great LA:

Download HDD and SSD

Here, sda is a HDD in RAID (mdadm), sdc is an SSD operating as a cache. sdb is the second HDD from the raid, I didn’t give the schedule - everything is very similar to sda.
HDD boot:

As you can see, the HDD becomes very bad - the response time reaches 1 second.

SSD, everything is simple and clear here:

Site loading time

Measured the loading time of the main page . There are several SSI with block caching. This is my favorite schedule:



In my opinion, the results are obvious.

Results and pitfalls

Our server has become much more responsive, MySQL is faster (measurements were not made), and hair is softer and silky.

If your server is barely breathing due to the heavy load on i / o - do not think. Put flashcache. On the project page there is a good description of how to prepare it and with what. I will not dwell on this - there is documentation for this. I just want to highlight some of the difficulties I encountered:

• Flashcache does not work on 32-bit systems. There is some kind of stupid mistake in the code that no one hurries to fix - Facebook has all the environment 64-bit.
• Included are hooks and scripts in initramfs to initialize flashcache volumes on Debian at startup, but it does not work for volumes on top of lvm (because at the time of launch lvm is not yet initialized). I decided to hack - / sbin / vgchange -ay at the beginning of the script. Wrong, of course, but then it was necessary to do it quickly.
• It is strictly forbidden to mount and use a volume over which flashcache runs (even if it is disabled). Well, that is, of course, possible, but then flashcache will not know about the changes. For me, it ended up with a slightly beaten FS after a restart (the third failure on the charts). That's right - stop flashcache, disable it (dmsetup remove), do everything you need with the volume, remove the flashcache volume on the SSD (flashcache_destroy), create the flashcache again. For operations like checking fs, all this is not necessary to do - just check flashcache.