About the performance of Thin Provision in LVM2

Since RHEL 6.4, thin provisioning support has been included in LVM2. I would translate it into Russian as a “subtle reservation”, although the translation is inaccurate and completely inconsistent with reality, therefore, the English spelling will be used along with the Russian.

Thin provisioning is the creation of logical volumes that initially use a bit of space and grow as data is written to them. In ZFS, this has been implemented a long time ago by the very philosophy of this file system. In VMware, this is used in every product. It came to LVM2, widely used in Linux today. Also one of the main innovations is thin snapshots, when for snapshots there is no need to reserve a place in advance, and it “grows” along with the changed data. Nested snapshots (snapshots of snapshots with any depth) are also allowed and encouraged, while declaring that there is no performance drop. A few nuances of use will be discussed in the article.

For stable operation, thin provision in LVM2 requires kernel 3.4. Red Hat is backported and works on their “classic” 2.6.32.
In close to me, Debian Wheezy thin provisioning is unavailable due to the lack of a key when compiling lvm2 - with-thin = internal and other difficulties. If necessary, for the purposes of the test, you can compile this package from source.
')
I was more interested not in the snapshots, but in the performance of “thin logical volumes” (Thin Logical Volume) for use on servers. For the impatient, I will say straight away - a drop in performance is observed, and a significant one. Details below.

On the same server with CentOS 6.4 2.6.32-279.2.1.el6.x86_64 on RAID1 made with the help of mdadm, we create a “thin pool” (thin pool):

lvcreate -L 50G -T /dev/VG/thin_pool 

and the usual logical volume (LV):

 lvcreate -L 50G /dev/VG/thick_vol 

Let's measure the performance of linear reading and writing to these devices:

Thin pool:

Record:

 dd if=/dev/zero of=/dev/mapper/VG-thin_pool bs=1M count=1000 oflag=direct 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 15.9126 s, 65.9 MB/s 

Reading:

 dd if=/dev/mapper/VG-thin_pool of=/dev/null bs=1M count=1000 iflag=direct 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 4.19247 s, 250 MB/s 

LV:

- Record:

 FIO: : iodepth=2, bw=943120 B/s, iops=230, avg=8645.48 usec 

 # dd if=/dev/zero of=/dev/VG/thick_vol bs=1M count=1000 oflag=direct 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 15.8432 s, 66.2 MB/s 

- Reading:

 FIO: : iodepth=2, bw=775490 B/s, iops=189, avg=10536.95 usec 

 # dd if=/dev/VG/thick_vol of=/dev/null bs=1M count=1000 iflag=direct 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 4.04925 s, 259 MB/s 

As you can see, the performance is the same for LV and thin pool.

Create a thin volume (thin logical volume) in the thin pool

 lvcreate -V 100G -T -n thin_vol /dev/mapper/VG-thin_pool 

Check performance:

Record:

 FIO: : iodepth=2, bw=1100.5KB/s, iops=275, avg=7241.55 usec 

 dd if=/dev/zero of=/dev/mapper/VG-thin_vol bs=1M count=1000 oflag=direct 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 38.7505 s, 27.1 MB/s 

Reading:

 FIO: : iodepth=256, bw=86100KB/s, iops=21524, avg=11860.53 usec 

 dd if=/dev/mapper/VG-thin_vol of=/dev/null bs=1M count=1000 iflag=direct 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 4.45363 s, 235 MB/s 

It is very well seen that the linear recording on the thin volume is 2 times slower than on a regular LV (27.1 MB / s against 66.2 MB / s). At the same time, the speed of random writing even increases, but for random reading it was not possible to measure the real performance at all - it shows only the level of reading from the cache, while resetting the cache did not help.

The drop in performance of the linear recording is alarming and there is a possibility that this is due to the presence of RAID1 from mdadm, so I had to test it on another machine, and also look at the performance at the file system level. VMware Workstation Debian Wheezy 7.3 3.10-0.bpo.3-amd64 SSD Disk.

Check performance:

LV:

Record:

 dd if=/dev/zero of=/delete.img bs=1M count=1000 oflag=direct 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 3.49623 s, 300 MB/s 

Reading:

 FIO: : iodepth=384, bw=158839KB/s, iops=39709, avg= 9.64 msec 

 dd if=/delete.img of=/dev/null bs=1M count=1000 iflag=direct 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 2.31352 s, 453 MB/s 

Thin LV:

Record:

 dd if=/dev/zero of=/mnt/thin/delete.img bs=1M count=1000 oflag=direct 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 7.15546 s, 147 MB/s 

Reading:

 FIO: : iodepth=384, bw=176574KB/s, iops=44143, avg=8675.36 usec 

 dd if=/mnt/thin/delete.img of=/dev/null bs=1M count=1000 iflag=direct 1000+0 records in 1000+0 records out 1048576000 bytes (1.0 GB) copied, 2.33435 s, 449 MB/s 

Random record performance could not be measured.
And again, we observe a drop in the speed of a linear recording twice, as in the previous test. Perhaps this is due to the nuances of the virtual machine.

UPDATE: Made another test on a “clean” server without RAID and virt.mashin. There is no drop in linear recording performance! I apologize to the distinguished audience for misleading the one-sided testing in the first two tests. As a result, I consider it necessary to test the performance for each specific case.

RESULTS:

1. When using thin volume in each particular case, it is necessary to perform linear recording performance testing, since, for example, for RAID 1 with mdadm and VMware virtual machines, the linear recording speed on it drops by 2 times compared to the “classic” LV;
2. At the same time, a random recording on a thin volume of such a drop did not even notice the speed at the same level as for LV;
3. On Debian, I’ll not use production in the meantime due to the lack of standard repositories;