We measure the performance of drives or again about IOPS

Inspired by the post respected amarao on how to measure disk performance.

Purpose:

Test the performance of the available means of storing information and make sure that the chosen methodology is correct, as well as understand the difference in performance between different types of drives, as well as enterprise-level and consumer-level hard drives.

Equipment:

1. SD card Sandisk Class 10 UHS 1 Extreme Pro 8 GB (up to 95 MB / s read, up to 90 MB / s write)
2. Team Class 10 32 GB SD card (up to 20 MB / s)
3. Transcend 2GB SD Card without Speed ​​Class
4. SSD OCZ-AGILITY3 60 GB
5. SATA-disk consumer-level Hitachi Deskstar HDS723020BLA642 2 TB 7200 rpm, 64 MB
6. Western Digital RE3 WD2502ABYS-23B7A0 enterprise-level SATA disk 250 GB 7200 rpm 16 MB
7. SATA-disk consumer-level Seagate Barracuda 7200.11 ST3320613AS 320 GB 7200 rpm 16 Mbyte
8. CD-ROM
9. Ramdisk / dev / ram on linux

')

Testing method:

The technique is fully described in the post . There are some somewhat unclear points:

We select such a depth of parallelism of operations that the latency remains within reasonable limits.

The task is to choose an iodepth so that avg.latency is less than 10ms.

Since the testing does not use storage systems and SAS disks, but various SATA drives, we don’t have to measure parallelism.
It is very time consuming to clean the disk before each test (dd if = / dev / zero of = / dev / sdz bs = 2M oflag = direct), so we will do this before testing once for each drive.
Testing the entire disk is completely time consuming, so we will use testing for 30 seconds.
So, we formulate the testing methodology for our case:
Get the value of IOPS, issued by the drive with random reading and writing in blocks of 4 Kbytes and avg.latency delay of no more than 10 ms during the test in 30 seconds . Also for the sake of completeness, let's measure the linear recording speed.

Fio read configuration:

  [readtest]
 blocksize = 4k
 filename = / dev / sda
 rw = randread
 direct = 1
 buffered = 0
 ioengine = libaio
 iodepth = 1024

 runtime = 30 

Fio configuration to write:

  [writetest]
 blocksize = 4k
 filename = / dev / sdz
 rw = randwrite
 direct = 1
 buffered = 0
 ioengine = libaio
 iodepth = 1024

 runtime = 30 

Results:

SD card Sandisk Class 10 UHS 1 Extreme Pro 8 GB (up to 95 MB / s read, up to 90 MB / s write)

Measurement of sequential reading:

  root @ host: ~ # dd if = / dev / sdb of = / dev / null iflag = direct bs = 2M
 3790 + 0 records in
 3790 + 0 records out
 7948206080 bytes (7.9 GB) copied, 419.551 s, 18.9 MB / s 

Reading analysis:

When reading, a pattern is clearly observed: the larger the queue size, the greater the IOPS and the greater the delay. Accordingly, the drive performance for reading: iodepth = 8, bw = 4020.6KB / s, iops = 1005, avg = 7.95 msec .










Measurement of sequential write:

  root @ host: ~ # dd if = / dev / zero of = / dev / sdb oflag = direct bs = 1M  
 dd: writing `/ dev / sdb ': No space left on device
 7581 + 0 records in
 7580 + 0 records out
 7948206080 bytes (7.9 GB) copied, 555.815 s, 14.3 MB / s 

Record Analysis:

Recording is different. The larger the queue size, the lower the speed in MB / s and IOPS, and the higher the delay. Accordingly, the write performance of the drive: iodepth = 1, bw = 618686 B / s, iops = 151, avg = 6.60 msec.
With sequential recording, it was not even possible to get close to the 90 MB / s promised by the manufacturer for writing - the maximum speed is 14.3 MB / s. The difference in speed between arbitrary and linear recording is palpable - more than 6 times.

Team Class 10 32 GB SD card (up to 20 MB / s)

Measurement of sequential write:

 root @ host: ~ # dd if = / dev / zero of = / dev / sdb oflag = direct bs = 2M
dd: writing `/ dev / sdb ': No space left on device
15600 + 0 records in
15599 + 0 records out
32714522624 bytes (33 GB) copied, 2212.11 s, 14.8 MB / s 

Record Analysis:

Under no circumstances was it possible to squeeze more than 1 IOPS on the record. Accordingly, the write performance of the drive does not meet our requirements for avg.latency <10 ms, since its performance is: iodepth = 1, bw = 5960 B / s, iops = 1, avg = 687.15 msec . Although the sequential recording drive actually corresponds to Class 10 - 14.8 MB / s.

Transcend 2GB SD Card

Measurement of sequential reading:

 root @ host: ~ # dd if = / dev / sdb of = / dev / null iflag = direct bs = 2M
957 + 1 records in
957 + 1 records out
2007498752 bytes (2.0 GB) copied, 195.504 s, 10.3 MB / s 

Reading analysis:

The linear reading speed corresponds to Class 10, although the flash drive was bought for a very long time and was “unpretentious”. Drive performance for reading: iodepth = 4, bw = 2447.4KB / s, iops = 611, avg = 6.53 msec .









Measurement of sequential write:

 root @ host: ~ # dd if = / dev / zero of = / dev / sdb oflag = direct bs = 2M
dd: writing `/ dev / sdb ': No space left on device
958 + 0 records in
957 + 0 records out
2007498752 bytes (2.0 GB) copied, 392.872 s, 5.1 MB / s 

Record Analysis:

Failed to get specified latency from drive. An interesting fact is that regardless of the queue length, the IOPS value was constant - 8. Serial writing to the media is slower than reading and does not correspond to Class 10. The closest level: iodepth = 1, bw = 35230 B / s, iops = 8, avg = 116.23 msec .

SSD OCZ-AGILITY3 60 GB

Measurement of sequential reading:

 root @ host # dd if = / dev / sdd of = / dev / null iflag = direct bs = 2M 
28620 + 1 records in
28620 + 1 records out
60022480896 bytes (60 GB) copied, 238.535 s, 252 MB / s 

Reading analysis:

The pattern is also visible: the larger the queue size, the greater the IOPS and the greater the delay. Accordingly, the drive performance for reading: Reading: iodepth = 64, bw = 52705KB / s, iops = 13176, avg = 4849.37 usec . The measured linear reading speed of 252 MB / s is far below the declared by the manufacturer (Max Read: up to 525MB / s).













Measurement of sequential write:

 root @ host # dd if = / dev / zero of = / dev / sdd oflag = direct bs = 2M
dd: writing `/ dev / sdd ': No space left on device
28621 + 0 records in
28620 + 0 records out
60022480896 bytes (60 GB) copied, 238.875 s, 251 MB / s 

Record Analysis:

It should be noted non-linear pattern of IOPS from avg.latency delays. Entry: iodepth = 128, bw = 70591KB / s, iops = 17647, avg = 7244.71 usec . The measured linear write speed of 251 MB / s is far below the declared by the manufacturer (Max Write: up to 500MB / s).

UPDATE : As suggested amarao SSD recording testing should be carried out for a longer period of time. Therefore, an additional test for 10 hours was launched. The iodepth number is 64, since in a shorter period of time this queue length turned out to be the most efficient. The configuration file is:

As a result, IOPS and bw decreased by more than 2 times:



Since fio does not allow to see the dependence of indicators on the time "out of the box" - it was not possible to estimate at the expiration of how long the performance began to fall.

SATA-disk consumer-level Hitachi Deskstar HDS723020BLA642 2 TB 7200 rpm, 64 MB

Measurement of sequential reading:

  root @ host # dd if = / dev / sdb of = / dev / null iflag = direct bs = 2M
 953864 + 1 records in
 953864 + 1 records out
 2000398934016 bytes (2.0 TB) copied, 15221 s, 131 MB / s 

Reading analysis:

The linear read speed is very high - 131 MB / s. Could not achieve the desired level avg.latency. The closest results: iodepth = 1, bw = 258021 B / s, iops = 62, avg = 15.85 msec .

Record Analysis:

The record, surprisingly, turned out to be better. It was possible to achieve 141 IOPS at 7 ms latency. iodepth = 1, bw = 580929 B / s, iops = 141, avg = 7025.40 usec .

Western Digital RE3 WD2502ABYS-23B7A0 enterprise-level SATA disk 250 GB 7200 rpm 16 MB

Measurement of sequential reading:

 root @ host: ~ # dd if = / dev / sda of = / dev / null iflag = direct bs = 2M
119237 + 1 records in
119237 + 1 records out
250059350016 bytes (250 GB) copied, 2603.86 s, 96.0 MB / s 

Reading analysis:

The linear read speed is lower than the previous consumer-level drive. Could not get result for given avg.latency level <10 ms. The closest reading result: iodepth = 1, bw = 349284 B / s, iops = 85, avg = 11.69 msec .









Measurement of sequential write:

 root @ host: ~ # dd if = / dev / zero of = / dev / sda oflag = direct bs = 2M   
dd: writing `/ dev / sda ': No space left on device
119238 + 0 records in
119237 + 0 records out
250059350016 bytes (250 GB) copied, 2615.61 s, 95.6 MB / s 

Record Analysis:

It should be noted the stability of IOPS on the size of the queue throughout the test. Drive characteristics: iodepth = 1, bw = 768989 B / s, iops = 187, avg = 5294.30 usec .

SATA-disk consumer-level Seagate Barracuda 7200.11 ST3320613AS 320 GB 7200 rpm 16 Mbyte

Reading analysis:

Could not achieve the desired level of avg.latency. The closest results: iodepth = 1, bw = 227153 B / s, iops = 55, avg = 18.00 msec .

Record Analysis:

As it has already become a regularity, things are better with recording to disk. It was possible to achieve 105 IOPS at 9.5 ms latency. iodepth = 1, bw = 430597 B / s, iops = 105, avg = 9477.07 usec .

CD-ROM

There is no need to test in detail, as the numbers are needed more for orientation. Drive did not allow avg.latency <10 ms condition to be met. The closest reading result: iodepth = 1, bw = 23655 B / s, iops = 5 .

Ramdisk / dev / ram on linux

Reading analysis:

It seems that everything is correct, the measurement of reading a RAM disk corresponds to the measured one , but there is one nuance:

 cpu: usr = 34.51%, sys = 64.60%, ctx = 1, majf = 0, minf = 0 

in other words, we rested against processor performance. And not the fact that the RAM disk itself. The result of reading at the moment: iodepth = 2, bw = 852105KB / s, iops = 213026, avg = 5.51 usec .

Record Analysis:

The same with recording on a ramdisk. We rested in the processor:

 cpu: usr = 20.86%, sys = 76.26%, ctx = 1, majf = 0, minf = 0 

The result of the recording at the moment: iodepth = 2, bw = 693857KB / s, iops = 173464, avg = 6.44 usec

Summary table of test results:

Appeared questions:

1. How can you imagine the size of the queue in real life and what does it say about the carrier? For example, on an SD card, queue size is 1, and on SSD disks, 64.
2. Is 30 seconds enough for an IOPS test or is it necessary to test the entire disk?
3. RAM- CPU. ?

:

1. . .
2. UPDATE: 30 RAM-, IOPS. SSD- .
3. .
4. SSD- . IOPS , latency.
5. enterprise consumer IOPS Latency. — . .
6. The linear speed of enterprise disks is no higher than consumer disks
7. Hard drives give a very low speed when read randomly. No more than half a megabyte per second.
8. IOPS read performance on measured hard drives was lower than on write. This is probably done in order to “equalize” the performance of reading and writing when taking into account RAID penalties, while the low reading IOPS is compensated for by the OS built-in cache.
9. The linear speed of the tested disks is arbitrary higher by up to 500 times or more.

PS I would like more results of testing enterprise-level drives, but it was no longer at hand.

Have a nice day!