📜 ⬆️ ⬇️

Comparative testing of software RAID performance. Linux vs FreeBSD (mdadm vs gmirror)

Introduction
Opinion with regards to various VS I have long been formed - it all depends on the tasks. But no, no, yes, there is a desire to dig deeper, to find out who is still stronger - Bruce Lee or Jackie Chan, Stallone or Schwarzenegger, mdadm or gmirror.
The test does not claim absolute objectivity; rather, it is even subjective in terms of the hardware used. But anyway, numbers are numbers.
Who are interested, please come under cat.


Test stand
Motherboard: ECS 865GV-M
Processor: Intel® Celeron® CPU 2.00GHz
HDD 1 (system): SATA-II 160Gb Western Digital WD1600AAJS (connected to mother)
HDD 2.3 (RAID): SATA-II 1.5Tb Western Digital [WD15EARS] (connected to the Promise)
SATA controller: Promise SATA300 TX4 PCI, 4-port SATA300
RAM: 1280Mb
* God knows what configuration, but, as they say, the rich, and so happy.

Start
Information about the utilities used in the test:
Bonnie ++
As the name implies, Bonnie ++ is based on the Bonnie benchmark. There are many different types of file operations and different programs use them in different ways. Bonnie ++ tests some of them and displays the result for each of the tests. Distributed under the GPL2 license.
There are two sections of operations in tests. The first is the IO test, which simulates the behavior of some database applications. The second is a test for creating, reading and deleting a large number of small files.
Website: www.coker.com.au/bonnie++
Iozone
IOzone is a console utility for measuring file system speed. The speed of such operations as read, write, re-read, re-write, read backwards, read strided, fread, fwrite, random read, pread, mmap, aio_read, aio_write is tested. The results are output to a file, the data from which can be used to plot graphs in Excel.
Website: www.iozone.org
DD
I think that the reader is familiar with this wonderful tool, so I will limit myself to quoting the wiki.
"Dd (English dataset definition - the definition of a set of data) - a Unix program designed to both copy and convert files"
')
Test OS:
1.FreeBSD 8.0 RELEASE i386
2.Debian Lenny 5.0.5 i386

Training
FreeBSD:

#gmirror load

#gmirror label -v gm0 /dev/ad6 /dev/ad10
UFS + Soft Updates
#newfs -U /dev/mirror/gm0

#mkdir /raid

#mount /dev/mirror/gm0 /raid

Linux:

debian:/home/iam# mdadm --create /dev/md0 --level=1 --raid-devices=2 /dev/sd[bc]1
mdadm.conf
debian:/home/iam# mdadm --detail --scan >> /etc/mdadm/mdadm.conf
EXT3
debian:/home/iam#mkfs.ext3 /dev/md0

debian:/home/iam#mkdir /raid

debian:/home/iam#mount /dev/md0 /raid


Let's go (with)
The first to go into battle is bonnie ++, we start with the default settings.
# bonnie++ -d /raid/ -m 'FREEBSD_gmirror' -u root > out.csv && cat out.csv | bon_csv2html > gmirror.html
About keys:
-d: set the program in which directory to create temporary files
-m: report header
-u: from whom to run
Then I redirected the output of the program to the file> out.csv, then with the help of bon_csv2html translated into readable html.
Results:
freebsd
image
linux
image
freebsd
image
linux
image

* I confess, in Linux I didn’t get readable html, so I fixed the result of html from FreeBSD
In the screenshots, we see the result of the team, let's go in order.
Size 2512M - RAM * 2
Sequential Output (Sequential Output) Sequential Input (Last Input)
Per char
These are the read results for a single byte. Demonstrate the ability of the OS to buffer such operations.
Block
read-write blocks of size 8192 bytes.
Rerite
I think does not need an explanation
Random Seeks (random reads)
Seeks - the number of random read operations
Sequential Create (sequential file creation)
Random Create (random / random file creation)
I think that now the reader does not understand only the lines filled with +++ characters - the test was completed so quickly that the program cannot display the result.

Now add some instructions to the program.
# bonnie++ -d /raid/ -m 'FREEBSD_gmirror' -n 100:16384:0:20 -u root > out.csv && cat out.csv | bon_csv2html > gmirror_1.html
-n100: 16384: 0: 20: speak to create 100 files from 0 to 16384 b in 20 subdirectories
Results:
freebsd
image
linux
image
freebsd
image
linux
image
* attentive reader will notice the difference in the versions of the program. Well, put the latest available in ports / repositories.

Waters:
As you can see the results of the program are broken for better perception. I will ignore the launch of the program with the default settings, focus on the last four screenshots.
Sequential Output
As can be seen in the “byte-byte” Per Char test, Linux flies to the spot so quickly that FreeBSD with its 169 KB / s does not even have time to go to the initial boot process. 22183 KB / s on Linux versus 169 KB / s on FreeBSD. CPU usage is minimal.
In a test where the record size is 8192 B, FreeBSD wins with almost a double margin. 48840 KB / s on FreeBSD versus 28895 KB / s on Linux. But FreeBSD loaded the CPU twice as much.
The rewrite test showed a slight difference, 23306 KB / s on FreeBSD versus 18529 KB / s on Linux. From which it can be concluded that the caching mechanisms of the tested OS are approximately equal in efficiency.
Sequential Input (Last Input)
Here, the victory is clearly for Linux. Per Char 350 KB / s on FreeBSD versus 26435 KB / s on Linux. Blocks in 8192 b, 67290 KB / s FreeBSD versus 86610 KB / s on Linux.
Random Seeks (random reads)
Linux wins this test, 165.1s on FreeBSD versus 148.5c on Linux.

Let's go to the last two screens.
Sequential Create (sequential creation)
In this test, you create / read / delete files sequentially. Let me remind the reader that we create 100 files in 20 subdirectories of arbitrary size from 0 to 16 KB.
Speed ​​is measured in files per second. Creating a FreeBSD 103 file / s against 3099 files / s for Linux, I also ask you to pay attention to the CPU load. Reading, 1556 files / s in FreeBSD versus 34204 files / s in Linux. Delete 12206 files / s from FreeBSD against 14,121 files / s from Linux. Linux, with a huge gap from the pursuer, breaks the finishing tape.
Random Create (arbitrary / random creation)
And here, Linux is the undisputed leader. Creating, FreeBSD 100 files / s against 20187 files / s for Linux. Reading, 166 files / s in FreeBSD versus 3299 files / s in Linux. Deleting 9139 files / s from FreeBSD against 11638 files / s from Linux

Summing up the results of bonnie ++, it can be noted that Linux, with some exceptions, won a landslide victory. Operations to create a large number of small files occur many times faster than in FreeBSD. But I advise the reader not to rush to the final conclusions, not everything is so obvious. Go ahead.

With bonnie ++ , we’ll finish this and move on to the next tool, Iozone .
Having played enough with various startup keys, having read the documentation, the following command was born:
# iozone -Raz -b /home/iam/results_all.xls -f /raid/1 -g 1g
* In fact, I am cunning, I spied the team in the test results on the program website))
About keys:
-R: tell the program to create a report file
-a: fully automatic mode, the recording size is automatically selected from 4k to 16m (4k..8k..16k..32k .... 16384k)
-f: specify the file on our array
-g: maximum file size is 1Gb (for -a mode)
-z: forces the program started in automatic mode to use small recording sizes for large files (i.e. from 4k to 16m (4k..8k..16k..32k .... 16384k))

For clarity, I will give the results in graphs. With the specified keys, the program creates a pretty weighty report file, in the charts I highlighted the most interesting, in my opinion, results.
The first will be a test for writing.
image
image
image
The results shown in the first chart do not leave FreeBSD any chance. When creating a file of 16Mb in size from 4 kilobytes to 16 megabytes, the average Linux value is ~ 204 MB / s, FreeBSD 44.7 MB / s.
The file is 128 MB in size, and when the block size reaches 128KB, the picture changes dramatically and FreeBSD confidently takes the lead. The mean values ​​are: Linux ~ 47.8 MB / s, FreeBSD 48.3 MB / s.
The file size is 1Gb. FreeBSD maintains the same speed as the two previous files. Linux, on the contrary, slowed down a lot. The mean values ​​are: Linux ~ 33 MB / s, FreeBSD 48.5 MB / s.
As you can see with smaller files, Linux handles much faster, as we have already noticed. With a 1Gb file, FreeBSD takes revenge. It is also revealing that FreeBSD maintains the same speed regardless of file size.

Next on the list we are reading.
image
image
image
First we give the file the size of 16 Mb. The competitors go almost hull to hull.
Average values: Linux ~ 534 MB / s, FreeBSD ~ 538 MB / s.
File 128 Mb. From the start, Linux leaves with grinding, but already at the 64 KB checkpoint, his opponent catches up with him and a fierce struggle begins, the first time to finish is Linux. Average values: Linux ~ 515 MB / s, FreeBSD ~ 380 MB / s.
When they reach our heavyweight, the test OSs bite at the start.
File size of 1Gb. Linux leaving FreeBSD to swallow the smoke at the start goes into the gap, as you can see, the speed reaches 1Gb / s. After the breakpoint (starting at 128KB block size), FreeBSD keeps a little behind the leader. Average values: Linux ~ 553 MB / s, FreeBSD ~ 369 MB / s.
Linux reads much faster than FreeBSD. Regardless of file size.

Randomized (random) recording
image
image
image
File 16 Mb. The results speak for themselves, in Linux we see a smooth curve, FreeBSD from something goes to the pitstop after a checkpoint of 1Mb and barely falls short of the finish. Average results: Linux ~ 356.8 MB / s, FreeBSD ~ 255.5 MB / s.
File 128 Mb. FreeBSD obviously didn’t come across and shows depressing results, Linux to the finish also felt wrong, the difference between the block sizes of 8 Mb and 16 Mb is striking. Average results: Linux ~ 6 MB / s, FreeBSD ~ 3.8 MB / s.
File 1 Gb. As with the sequential creation of a large-sized file, FreeBSD confidently comes forward, Linux catches the opponent only to the finish. Average results: Linux ~ 12 MB / s, FreeBSD ~ 26 MB / s.

Random (random) reading
image
image
image
The first graph, Linux, was only slightly ahead, but was already overtaken by a 32 KB checkpoint. Next, the contenders moved flush, with very little FreeBSD benefits. Average results: Linux ~ 523 MB / s, FreeBSD ~ 458 MB / s.
The picture with the test results of the file size of 128 Mb differs very little from the first one, with the only difference that the clear leader, starting from the block size of 64 KB, was not identified. Average results: Linux ~ 476 MB / s, FreeBSD ~ 362 MB / s.
The last graph gives us a clear understanding that Linux is far in the forefront, when reading large files. Average results: Linux ~ 523 MB / s, FreeBSD ~ 274 MB / s.

Toward the close of the test results all known utility DD.
Create a 1 Gb file of 4 KB blocks
FreeBSD: 49.02 MB / s
Linux: 23.9 MB / s
We read this file in blocks of 4 KB
FreeBSD: 342.7 MB / s
Linux: 788 MB / s

Create a 1 Gb file of 64 KB blocks
FreeBSD: 48.64 MB / s
Linux: 35.0 MB / s
We read this file in blocks of 64 KB
FreeBSD: 350.14 MB / s
Linux: 617 MB / s

Create a 1 Gb file of 1 Mb blocks
FreeBSD: 47.98 MB / s
Linux: 23.9 MB / s
We read this file in blocks of 4 KB
FreeBSD: 348.51 MB / s
Linux: 460 MB / s

Create a file of 1000 Mb in size from blocks of 100 Mb
FreeBSD: 44.24 MB / s
Linux: 37.7 MB / s
We read this file in blocks of 4 KB
FreeBSD: 308.61 MB / s
Linux: 459 MB / s

Create a file of 100 Gb in size from 1 Gb blocks
FreeBSD: 31.7 MB / s
Linux: 14.7 MB / s
We read this file in blocks of 4 KB
FreeBSD: 57.48 MB / s
Linux: failed to get result

FreeBSD is clearly faster than Linux when creating files. Linux is more powerful in reading files.

Listings dd
Freebsd
* dd output to FreeBSD I did a little tweaking for readability
1Gb 4k
# /usr/bin/time -h dd if=/dev/zero of=testfile bs=4k count=262144
262144+0 records in
262144+0 records out
1073741824 bytes transferred in 20.888121 secs (49,02 Mbytes/sec)
21.05s real 0.45s user 11.12s sys
4k
*
# /usr/bin/time -h dd if=testfile of=/dev/null bs=4k count=262144
262144+0 records in
262144+0 records out
1073741824 bytes transferred in 2.987966 secs (342,7 Mbytes/sec)
2.99s real 0.15s user 2.82s sys

# /usr/bin/time -h dd if=/dev/zero of=testfile bs=64k count=16384
16384+0 records in
16384+0 records out
1073741824 bytes transferred in 21.050771 secs (48,64 Mbytes/sec)
21.05s real 0.08s user 8.12s sys
# /usr/bin/time -h dd if=testfile of=/dev/null bs=64k count=16384
16384+0 records in
16384+0 records out
1073741824 bytes transferred in 2.924540 secs (350,14 Mbytes/sec)
2.92s real 0.05s user 2.86s sys

# /usr/bin/time -h dd if=/dev/zero of=testfile bs=1m count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 21.341423 secs (47,98bytes/sec)
21.57s real 0.00s user 7.87s sys

# /usr/bin/time -h dd if=testfile of=/dev/null bs=1m count=1024
1024+0 records in
1024+0 records out
1073741824 bytes transferred in 2.938155 secs (348,51 Mbytes/sec)
2.94s real 0.00s user 2.92s sys

# /usr/bin/time -h dd if=/dev/zero of=testfile bs=100m count=10
10+0 records in
10+0 records out
1048576000 bytes transferred in 22.601393 secs (44,24 Mbytes/sec)
22.86s real 0.00s user 7.89s sys

# /usr/bin/time -h dd if=testfile of=/dev/null bs=100m count=10
10+0 records in
10+0 records out
1048576000 bytes transferred in 3.240300 secs (308,61 Mbytes/sec)
3.26s real 0.00s user 3.25s sys

# /usr/bin/time -h dd if=/dev/zero of=testfile bs=1g count=100
100+0 records in
100+0 records out
107374182400 bytes transferred in 3229.788959 secs (31,7 Mbytes/sec)
53m50.33s real 0.00s user 14m40.89s sys
# /usr/bin/time -h dd if=testfile of=/dev/null bs=1g count=100
100+0 records in
100+0 records out
107374182400 bytes transferred in 1781.360638 secs (57,48 Mbytes/sec)
29m41.67s real 0.00s user 6m44.73s sys


Linux
debian:/raid# dd if=/dev/zero of=testfile bs=4k count=262144

262144+0

262144+0

1073741824 (1,1 GB), 45,0159 c, 23,9 MB/c

debian:/raid# dd if=testfile of=/dev/null bs=4k count=262144

262144+0

262144+0

1073741824 (1,1 GB), 1,36295 c, 788 MB/c

debian:/raid# dd if=/dev/zero of=testfile bs=64k count=16384

16384+0

16384+0

1073741824 (1,1 GB), 28,2238 c, 38,0 MB/c

debian:/raid# dd if=testfile of=/dev/null bs=64k count=16384

16384+0

16384+0

1073741824 (1,1 GB), 1,73935 c, 617 MB/c

debian:/raid# dd if=/dev/zero of=testfile bs=1M count=1024

1024+0

1024+0

1073741824 (1,1 GB), 30,711 c, 35,0 MB/c

debian:/raid# dd if=testfile of=/dev/null bs=1M

1024+0

1024+0

1073741824 (1,1 GB), 2,33447 c, 460 MB/c

debian:/raid# dd if=/dev/zero of=testfile bs=100M count=10

10+0

10+0

1048576000 (1,0 GB), 27,8344 c, 37,7 MB/c

debian:/raid# dd if=testfile of=/dev/null bs=100M count=10

10+0

10+0

1048576000 (1,0 GB), 2,28437 c, 459 MB/c

debian:/raid# dd if=/dev/zero of=testfile bs=1024M count=100

100+0

100+0

107374182400 (107 GB), 7315,88 c, 14,7 MB/c

debian:/raid# dd of=testfile if=/dev/null bs=1024M count=100

0+0

0+0

0 (0 B), 7,5694e-05 c, 0,0 kB/c


Epilogue
The idea to test, as I already wrote, was born spontaneously. Initially, I just wanted to assemble Software RAID for backups on a budget machine, and then this VS , his mother , got into it. For me, the choice is obvious - FreeBSD, it is a bit faster when creating files, especially when creating large files. Just what I need. Having carried out this test, I was only once again convinced of the correctness of the occupied position with regards to different holly war - it all depends on the tasks.
Although I’m not speaking the whole truth again. The choice in favor of FreeBSD was made before the test. Since I just know this OS much better than Linux, and I like it a lot more. If we look at the results objectively, then I, absolutely deservedly, will give the victory to Linux. Having run again according to the test results, there is no doubt about it.
Someone can fairly notice, but why did you “bother” so much with these tests, already bonnie ++ told you everything, but no. I used to approach everything thoughtfully, with feeling, sensibly. Yes, and the free time that appeared allowed to experiment.
How competent and informative my report is for you to decide, I am not a professional in * nix, my journey to this fascinating world began relatively recently.
Thanks to those who mastered.

PS
In the preparation of the material used:
1. Linux Administrator's Guide. Authors: Nemeth E., Snyder G., Trent H.
2. FreeBSD. Detailed guide. Michael Lucas.
3. google.com
4. / dev / hands and / dev / head =)
This report was prepared with the active help of comrades with unixadmins.su, a special thank you.

Source: https://habr.com/ru/post/98649/


All Articles