10Gbit network storage at reasonable cost
The task was the following - to upgrade the storage system for the photographer. Rather, photographers.
There was network storage on the basis of Windows Server 2008 (it happened historically), Adaptec 3405 RAID controller and 4x 1.5Tb disks connected to it. For communication with the outside world, the HP NC360T 2-port gigabit server network card was used.
Problems of the existing system:
- access speed to network resources "rested" in the performance of gigabit Ethernet interfaces,
- the place on the RAID array ended.
Wishes:
- quick network access to photo and video files located on the disk array,
- increase the available volume of the disk array by 2-3 times.
')
The idea of ​​combining gigabit interfaces into a trunk was tried and dropped as inoperative. Later communication with HP support and reading the standards once again confirmed that even the presence of multi-port server network cards and a managed switch with LACP support do not allow the existing system to balance the load across multiple links in connection 1 server - 1 client.
The idea to make storage based on low-cost Infiniband cards was also not very inspiring - it was precisely network storage with access to files from several computers. Let not everyone at a particularly high speed (for some, a gigabit interface is suitable, for some multi-stream WiFi type n, and there, you see, and ac will start to put laptops in laptops).
There is, in fact, one option - 10G Ethernet. Which has one trouble - the price.
Take, for example, HP NC523SFP (nee Qlogic) and get a price of about $ 1000 for 1 card.
We add about the same to the transceiver and get ... Economically absurd price for this task we get.
The issue price played a significant role. The task was not to “make it cool for 3 kopecks,” but the branded corporate solutions with which I had to deal with for the last ... years, did not go through the budget either.
And I had to turn my eyes on ebay.
There were picked up
- two 10G cards from Myricom 10G-PCIE2-8B2-2S at the price of $ 200 per piece,
- two 10G SFP transceivers of an unknown Chinese manufacturer for $ 60 apiece,
- Adaptec 7805 RAID Controller with SATA / SAS cables for $ 400.
Probably, the network cards and the controller could be taken simpler - cheaper, if you wait - pick up. But there was an opportunity from the States ...
Everything, except transceivers, arrived quickly, and transceivers went from China. Slowly drove. But still, the Russian mail took them :-)
Considering that more than two workstations with a fast connection channel are not planned even in the future, we did without a 10G switch. While the transceivers were traveling, an attempt was made to connect machines with Myricom direct attach twinax cards with a Cisco cable with a length of 7 meters. The system did not start. Cards "raised" the interface and then it was "lowered". The support service from Myricom requested a bunch of diagnostics from the cards (good, the cards have such an opportunity), gave a lot of advice like “try this, and then syak”, but at the end summed up “change the cable”.
Another twinax cable was not at hand, so I had to wait for the transceivers.
They were successfully received, put into cards, and connected by a 10 meter long LC-LC patch cord (purchased at a nearby shop for 700r). Link "rose" instantly.
On tests through the microsoft NTttcp it turned out 9.5 - 9.6 gigit. When enabled on AIM cards (Adaptive Interrupt Moderation) with a default interrupt delay of 25 microseconds. This, according to those support, in some cases, unnecessarily loads the CPU, but gives less latency.
With AIM turned off, test performance dropped by 30%. Perhaps this test is not an indicator for real download, but so far left with AIM. Fortunately, there is plenty of processor power for this task (Intel i7 3770).
This test showed that at least obvious nonsense is not done, the cards work at full speed.
The disk array was compiled on 5 WD Red 3TB disks. In RAID5. The proposed RAID6 was not accepted on the basis of “everything is still copied to the cloud”. Indeed, the server is configured with a program from CrashPlan, and after a 20 megabit channel, everything briskly goes to the data center somewhere in Atlanta. We tried to recover the data (somehow the old version of the file was needed). Recovers quite cheerfully.
Going disk array day. and another few hours there were copied data from the old array. During the build-test array could begin to use, but the performance was not interesting.
After assembling and copying data, testing began. To begin with synthetic tests.
ATTO benchmark that during direct launch on the server, that when working with the client showed something strange.
Test from the server.
Test from the client machine
Rather, it measures the speed of access to the controller cache in the case of a local connection. And in the case of a network connection, it is generally not clear what.
In this case, the network interface is loaded at 100%.
HD Tune Pro also showed something strange. Remembering that, at best, from single disks of this series can be obtained in the region of 150 megabytes per second for reading and writing large files, it seems that we are again measuring the controller cache.
More interesting was the real work with images.
Feels like the speed of viewing photos from Lightroom has increased 4 times.
When opening and saving large panoramas in Photoshop (tiff without compression with a non-glued 3.5-GB layer-by-layer panorama), the system “rested” on the performance of Photoshop on the client machine. The file of the second 4 was opened, 10 seconds were recorded. Most of these 10 seconds, Photoshop “thought about his own” without giving out any network activity.
The first group of peaks is the reading of a picture, the last group with the interface loading almost up to 100% is the recording of a picture.
Feels like everything has become faster at times. Yes, and 12TB of disk space is enough for a while :-)
I do not consider myself to be an expert in high performance computing, and I will be happy with comments - wishes about this system.
Continuation of a story.
There was network storage on the basis of Windows Server 2008 (it happened historically), Adaptec 3405 RAID controller and 4x 1.5Tb disks connected to it. For communication with the outside world, the HP NC360T 2-port gigabit server network card was used.
Problems of the existing system:
- access speed to network resources "rested" in the performance of gigabit Ethernet interfaces,
- the place on the RAID array ended.
Wishes:
- quick network access to photo and video files located on the disk array,
- increase the available volume of the disk array by 2-3 times.
')
The idea of ​​combining gigabit interfaces into a trunk was tried and dropped as inoperative. Later communication with HP support and reading the standards once again confirmed that even the presence of multi-port server network cards and a managed switch with LACP support do not allow the existing system to balance the load across multiple links in connection 1 server - 1 client.
The idea to make storage based on low-cost Infiniband cards was also not very inspiring - it was precisely network storage with access to files from several computers. Let not everyone at a particularly high speed (for some, a gigabit interface is suitable, for some multi-stream WiFi type n, and there, you see, and ac will start to put laptops in laptops).
There is, in fact, one option - 10G Ethernet. Which has one trouble - the price.
Take, for example, HP NC523SFP (nee Qlogic) and get a price of about $ 1000 for 1 card.
We add about the same to the transceiver and get ... Economically absurd price for this task we get.
The issue price played a significant role. The task was not to “make it cool for 3 kopecks,” but the branded corporate solutions with which I had to deal with for the last ... years, did not go through the budget either.
And I had to turn my eyes on ebay.
There were picked up
- two 10G cards from Myricom 10G-PCIE2-8B2-2S at the price of $ 200 per piece,
- two 10G SFP transceivers of an unknown Chinese manufacturer for $ 60 apiece,
- Adaptec 7805 RAID Controller with SATA / SAS cables for $ 400.
Probably, the network cards and the controller could be taken simpler - cheaper, if you wait - pick up. But there was an opportunity from the States ...
Everything, except transceivers, arrived quickly, and transceivers went from China. Slowly drove. But still, the Russian mail took them :-)
Considering that more than two workstations with a fast connection channel are not planned even in the future, we did without a 10G switch. While the transceivers were traveling, an attempt was made to connect machines with Myricom direct attach twinax cards with a Cisco cable with a length of 7 meters. The system did not start. Cards "raised" the interface and then it was "lowered". The support service from Myricom requested a bunch of diagnostics from the cards (good, the cards have such an opportunity), gave a lot of advice like “try this, and then syak”, but at the end summed up “change the cable”.
Another twinax cable was not at hand, so I had to wait for the transceivers.
They were successfully received, put into cards, and connected by a 10 meter long LC-LC patch cord (purchased at a nearby shop for 700r). Link "rose" instantly.
On tests through the microsoft NTttcp it turned out 9.5 - 9.6 gigit. When enabled on AIM cards (Adaptive Interrupt Moderation) with a default interrupt delay of 25 microseconds. This, according to those support, in some cases, unnecessarily loads the CPU, but gives less latency.
With AIM turned off, test performance dropped by 30%. Perhaps this test is not an indicator for real download, but so far left with AIM. Fortunately, there is plenty of processor power for this task (Intel i7 3770).
This test showed that at least obvious nonsense is not done, the cards work at full speed.
The disk array was compiled on 5 WD Red 3TB disks. In RAID5. The proposed RAID6 was not accepted on the basis of “everything is still copied to the cloud”. Indeed, the server is configured with a program from CrashPlan, and after a 20 megabit channel, everything briskly goes to the data center somewhere in Atlanta. We tried to recover the data (somehow the old version of the file was needed). Recovers quite cheerfully.
Going disk array day. and another few hours there were copied data from the old array. During the build-test array could begin to use, but the performance was not interesting.
After assembling and copying data, testing began. To begin with synthetic tests.
ATTO benchmark that during direct launch on the server, that when working with the client showed something strange.
Test from the server.
Test from the client machine
Rather, it measures the speed of access to the controller cache in the case of a local connection. And in the case of a network connection, it is generally not clear what.
In this case, the network interface is loaded at 100%.
HD Tune Pro also showed something strange. Remembering that, at best, from single disks of this series can be obtained in the region of 150 megabytes per second for reading and writing large files, it seems that we are again measuring the controller cache.
More interesting was the real work with images.
Feels like the speed of viewing photos from Lightroom has increased 4 times.
When opening and saving large panoramas in Photoshop (tiff without compression with a non-glued 3.5-GB layer-by-layer panorama), the system “rested” on the performance of Photoshop on the client machine. The file of the second 4 was opened, 10 seconds were recorded. Most of these 10 seconds, Photoshop “thought about his own” without giving out any network activity.
The first group of peaks is the reading of a picture, the last group with the interface loading almost up to 100% is the recording of a picture.
Feels like everything has become faster at times. Yes, and 12TB of disk space is enough for a while :-)
I do not consider myself to be an expert in high performance computing, and I will be happy with comments - wishes about this system.
Continuation of a story.
Source: https://habr.com/ru/post/181822/
All Articles