Testing Cloud Provider Drives
After launching a new storage system in Scalaxi , we migrated all user data from the old system to it and decided to compare the speed of the new system with the existing solutions on the market. Under the cat , the storage performance test of the following cloud providers: Amazon , Rackspace , Terremark , Scalaxi , Selectel , Clodo .
For those readers who want to immediately find out the result without plunging into the testing methodology, the summary is as follows: the Scalaxi storage system is generally the most productive among the tested providers, but the worst results were shown by the Clodo storage system.
Well, now more.
The initial list of companies was as follows: Amazon , Rackspace , Terremark , Scalaxi , Selectel , Clodo , Activecloud , VMco , IT-Grad vCloud , Slidebar.ru .
')
But, some of them fell off due to the following reasons:
Activecloud - the lack of hourly pay for resources, which in our opinion is a mandatory requirement.
VMco and IT-Grad vCloud are Russian representatives of the VMware platform and NetApp storage systems , the lack of hourly wages and the manual creation of machines.
Slidebar.ru - cloud hosting on the Microsoft Hyper-V platform, we couldn’t install Linux here during the week, because the ISO image mount function didn’t work.
Commenting on the rest.
Scalaxi - ourselves, IBRAID storage on the Infiniband bus, a solution based on Xen .
Foreign:
Amazon is the leader in the global cloud computing market, sets the bar for others, tested the fastest EBS disks (which everyone complains about anyway, since the loaded database works with difficulty), a solution based on Xen.
Rackspace is also a fairly well-known worldwide hoster that uses local drives in servers, instead of network storage, a Xen-based solution.
Terremark is a fairly large telecommunications company, the main difference is the VMware platform and high-performance professional EMC storage systems.
There are not so many honest Russian cloud hosters, so we took them all:
Selectel - colleagues working with iSCSI storage, a solution based on Xen.
Clodo - colleagues working with the first version of our 2008 architecture, IBM GPFS storage, Xen solution.
Here, of course, we could, with the help of our colleague, paint a long and thick matan , but limited ourselves to a simpler and more practical way.
For each provider, including ourselves, we created three virtual machines with debian 6 (or ubuntu 10.10, in case of its absence), 15 GB disks and 512 MB of RAM (excluding Amazon, their micro instance contains 613 MB), so that the cache in RAM did not greatly affect the results.
On all three machines, we simultaneously launched a test script that sequentially launches iozone , recognized as the "killer" of disks, successively three times in a row, which the disk manufacturers themselves are afraid of. We started it with the following parameters:
Further, the test results were collected , entered and averaged in the Google table . Thus, each test item was performed 9 times (3 times on each of the 3 virtual machines). The exception was Clodo, due to poor performance for the week we managed to get rid of only 1 test on each machine.
Tests were launched at 12 noon, a few hours before the peak load in Moscow. In general, testing in most cases lasted 2-3 days.
Previously, we checked for each provider that they do not use any shaper limiting disk operations. To do this, we created machines of small, medium and large sizes and compared the speed of unbuffered recording and reading. Many of them also take some money for disk operations, so there is no point in limiting them. In our case, we completely turned off the shaping of the discs, and turn them on only after we make it possible to individually adjust the band for each disc.
Separately, we should add that on our part two nodes of our storage, completely filled with real data, had a useful capacity of 20 TB and a constant load of 10,000 IOPS for reading and writing, in order not to test just the network speed and caches. Client profile - 90% of web projects of small and medium traffic (up to 50,000 unique visitors per day). We can only judge about the volume and occupancy of the repositories of other providers.
To study the results in more detail, we suggest reading the public Google document .
And here are static pictures with comments.
Primarily arbitrary read and write. In accordance with the considerations of Intel engineers , the standard database load profile is 67% of random reads, 33% of random entries in blocks from 2 to 8 KB. In general, real systems are in most cases arbitrary write and random read, so these tests are the most important.
It is worth adding that a regular SATA disk provides 120 random IOPS (operations per second) for reading and 60 arbitrary IOPS for writing, a SAS disk - 250 arbitrary IOPS for reading and 500 arbitrary IOPS for writing (due to write caching).
Scalaxi, Terremark and Selectlec cache data, so they generally have a higher reading than the rest. Selectel uses 1 Gbit / s Ethernet, and Terremark and Scalaxi, 8 Gbit / s FibreChannel and 40 Gbit / s Infiniband , respectively. Clodo in theory also uses caching, but IBM GPFS on random access annihilates all the advantage.
At the same time, obviously, Terremark, EMC or VMware have some problems with random reading in small blocks.
The cached entry again leads Scalaxxi, Terremark and Selectel to leaders. Amazon and Rackspace show the maximum possible speed of random writing to an array of SAS disks. Clodo's IBM GPFS is not designed for arbitrary operations and shows the worst results.
Terremark in this test as a whole surpassed our storage system, but we got the impression that the nodes of their storage system were not fully occupied and less loaded.
In this test, the Scalaxi shows the best results with a steady 300 MB / s due to the efficiency of its caching system and the readahead algorithm, which is closest to us in Rackspace speed, shows essentially the speed of reading from an array of local disks. Clodo is on a par with the rest, since GPFS caches data for reading and more or less works with sequential reading.
And again Terremark takes the lead. The value of Scalaxi is close to the results of Rackspace, namely the sequential writing to an array of local disks.
Considering that the load in Internet projects is mainly formed by two things - a database and distribution of small pictures and other files, our solution can be called ideal and as productive as possible for such projects.
Terremark need to adjust settings and everything will be on top.
Amazon’s strategy today is “quantity,” not “quality.”
Selectel is a good solution for ethernet and iSCSI.
Rackspace are local drives, which you say, but virtual servers are tightly bound to a physical server and any change can cause a long offline migration.
Clodo is too slow to use on real applications.
Continue to follow our blog and @ twitter , as soon:
In addition, as always, a week of free testing at registration .
For those readers who want to immediately find out the result without plunging into the testing methodology, the summary is as follows: the Scalaxi storage system is generally the most productive among the tested providers, but the worst results were shown by the Clodo storage system.
Well, now more.
Selection of companies
The initial list of companies was as follows: Amazon , Rackspace , Terremark , Scalaxi , Selectel , Clodo , Activecloud , VMco , IT-Grad vCloud , Slidebar.ru .
')
But, some of them fell off due to the following reasons:
Activecloud - the lack of hourly pay for resources, which in our opinion is a mandatory requirement.
VMco and IT-Grad vCloud are Russian representatives of the VMware platform and NetApp storage systems , the lack of hourly wages and the manual creation of machines.
Slidebar.ru - cloud hosting on the Microsoft Hyper-V platform, we couldn’t install Linux here during the week, because the ISO image mount function didn’t work.
Commenting on the rest.
Scalaxi - ourselves, IBRAID storage on the Infiniband bus, a solution based on Xen .
Foreign:
Amazon is the leader in the global cloud computing market, sets the bar for others, tested the fastest EBS disks (which everyone complains about anyway, since the loaded database works with difficulty), a solution based on Xen.
Rackspace is also a fairly well-known worldwide hoster that uses local drives in servers, instead of network storage, a Xen-based solution.
Terremark is a fairly large telecommunications company, the main difference is the VMware platform and high-performance professional EMC storage systems.
There are not so many honest Russian cloud hosters, so we took them all:
Selectel - colleagues working with iSCSI storage, a solution based on Xen.
Clodo - colleagues working with the first version of our 2008 architecture, IBM GPFS storage, Xen solution.
Testing method
Here, of course, we could, with the help of our colleague, paint a long and thick matan , but limited ourselves to a simpler and more practical way.
For each provider, including ourselves, we created three virtual machines with debian 6 (or ubuntu 10.10, in case of its absence), 15 GB disks and 512 MB of RAM (excluding Amazon, their micro instance contains 613 MB), so that the cache in RAM did not greatly affect the results.
On all three machines, we simultaneously launched a test script that sequentially launches iozone , recognized as the "killer" of disks, successively three times in a row, which the disk manufacturers themselves are afraid of. We started it with the following parameters:
iozone -a -i0 -i1 -i2 -O -s8G , which means to perform testing for sequential and random write / read data blocks from 4 KB to 16 MB on a file of 8 GB.Further, the test results were collected , entered and averaged in the Google table . Thus, each test item was performed 9 times (3 times on each of the 3 virtual machines). The exception was Clodo, due to poor performance for the week we managed to get rid of only 1 test on each machine.
Tests were launched at 12 noon, a few hours before the peak load in Moscow. In general, testing in most cases lasted 2-3 days.
Previously, we checked for each provider that they do not use any shaper limiting disk operations. To do this, we created machines of small, medium and large sizes and compared the speed of unbuffered recording and reading. Many of them also take some money for disk operations, so there is no point in limiting them. In our case, we completely turned off the shaping of the discs, and turn them on only after we make it possible to individually adjust the band for each disc.
Separately, we should add that on our part two nodes of our storage, completely filled with real data, had a useful capacity of 20 TB and a constant load of 10,000 IOPS for reading and writing, in order not to test just the network speed and caches. Client profile - 90% of web projects of small and medium traffic (up to 50,000 unique visitors per day). We can only judge about the volume and occupancy of the repositories of other providers.
results
To study the results in more detail, we suggest reading the public Google document .
And here are static pictures with comments.
Primarily arbitrary read and write. In accordance with the considerations of Intel engineers , the standard database load profile is 67% of random reads, 33% of random entries in blocks from 2 to 8 KB. In general, real systems are in most cases arbitrary write and random read, so these tests are the most important.
It is worth adding that a regular SATA disk provides 120 random IOPS (operations per second) for reading and 60 arbitrary IOPS for writing, a SAS disk - 250 arbitrary IOPS for reading and 500 arbitrary IOPS for writing (due to write caching).
Random read
Scalaxi, Terremark and Selectlec cache data, so they generally have a higher reading than the rest. Selectel uses 1 Gbit / s Ethernet, and Terremark and Scalaxi, 8 Gbit / s FibreChannel and 40 Gbit / s Infiniband , respectively. Clodo in theory also uses caching, but IBM GPFS on random access annihilates all the advantage.
At the same time, obviously, Terremark, EMC or VMware have some problems with random reading in small blocks.
Random record
The cached entry again leads Scalaxxi, Terremark and Selectel to leaders. Amazon and Rackspace show the maximum possible speed of random writing to an array of SAS disks. Clodo's IBM GPFS is not designed for arbitrary operations and shows the worst results.
Terremark in this test as a whole surpassed our storage system, but we got the impression that the nodes of their storage system were not fully occupied and less loaded.
Sequential reading
In this test, the Scalaxi shows the best results with a steady 300 MB / s due to the efficiency of its caching system and the readahead algorithm, which is closest to us in Rackspace speed, shows essentially the speed of reading from an array of local disks. Clodo is on a par with the rest, since GPFS caches data for reading and more or less works with sequential reading.
Sequential write
And again Terremark takes the lead. The value of Scalaxi is close to the results of Rackspace, namely the sequential writing to an array of local disks.
Final summary
Considering that the load in Internet projects is mainly formed by two things - a database and distribution of small pictures and other files, our solution can be called ideal and as productive as possible for such projects.
Terremark need to adjust settings and everything will be on top.
Amazon’s strategy today is “quantity,” not “quality.”
Selectel is a good solution for ethernet and iSCSI.
Rackspace are local drives, which you say, but virtual servers are tightly bound to a physical server and any change can cause a long offline migration.
Clodo is too slow to use on real applications.
Afterword
Continue to follow our blog and @ twitter , as soon:
- separate network shaping ;
- separate shaping for disks;
- VMware based on Infiniband;
- file storage;
- and much more;
- including what you offer yourself .
In addition, as always, a week of free testing at registration .
Source: https://habr.com/ru/post/117260/
All Articles