How to build a storage system with rocket on a standard hardware? SDS RAIDIX hardware platform architecture
RAIDIX - software storage or SDS (Software Defined Storage), which allows you to build on the basis of standard server hardware reliable, productive and fault-tolerant data storage.
In this article we want to talk about what requirements for hardware makes RAIDIX, describe the deployment options for our SDS, give examples of hardware configurations of storage systems based on RAIDIX and possible areas of their application.
equipment requirements
Deploying SDS RAIDIX will require the following server hardware:
')
- 1-2 Intel Xeon processors of a suitable model and the required amount of RAM;
- one or more SAS HBA adapters for connecting internal and / or external disk baskets; hardware RAID controllers with RAIDIX are not compatible;
- one or more cache synchronization interfaces in a dual-controller configuration; There are several options: SAS, InfiniBand, Ethernet; interface duplication is possible; in single-controller configuration, these interfaces are not needed;
- interfaces for connecting to SAN and / or NAS: Ethernet, InfiniBand, FC; possible direct connection to hosts (clients) via SAS;
- interfaces for control traffic and heartbeat; supports the use of dedicated or shared with other types of traffic ports Ethernet; enough bandwidth from 100Mb / s; for heartbeat, it is recommended to use dedicated interfaces with direct connection between controllers, this is only necessary for the dual-controller configuration;
- any standard HDD SAS / SATA models, with no restrictions on volume, speed, form factor or manufacturer;
- server platform suitable for installing the above equipment.
To connect a large number of disks it is supposed to use external disk shelves connected via SAS. It is recommended to use internal and external disk shelves with support for hot-swappable disks.
There is a compatibility sheet, a list of recommended and tested equipment.
Deployment options
RAIDIX offers two deployment options: single- and dual-controller. In the first version, the RAIDIX software is installed on one physical server acting as a storage controller. The disks are combined into a fail-safe RAID, but the server itself and some of its components form single points of failure. This may be acceptable for non-critical tasks.
Dual-controller configuration involves installing RAIDIX software on two identical physical servers, each of which becomes a storage controller. These can be separate server platforms, or a single platform with two server nodes (cluster-in-a-box). Both controllers are physically connected to a single disk pool hosted on internal and external disk baskets. RAIDIX integrates two servers into a fail-safe active-active cluster, the controller cache is synchronized across dedicated interfaces.
In normal mode, the load is evenly distributed to two controllers - half of the volumes created on the disk array are served by one controller, the other half - by the second controllers. If for some reason one of the controller nodes fails, the entire load automatically switches to the "surviving" controller without interruption and data loss. This solution eliminates the presence of single points of failure and is suitable for critical projects that are sensitive to downtime.
Dual-controller RAIDIX platform
AIC HA401-LB2 solution can be cited as a good example of a server platform for dual-controller RAIDIX storage configuration. This is a 4U platform for high-availability storage servers (cluster-in-a-box) with two identical server nodes, duplicated power supplies and a 24 HDD 3.5 ”internal disk cage with hot-swappable storage. Each server node (storage controller) supports two Intel Xeon processors, up to 2TB of RAM and 6 PCIe slots. Both server nodes are equipped with a pair of built-in 1GbE ports and a pair of built-in 10GbE ports. This is enough to deploy a very productive storage system with a capacity of from two dozen to several hundred disks. This platform can be called one of the recommended, it is successfully used in many projects based on RAIDIX.
A couple of built-in 1GbE ports can be conveniently used for storage management and heartbeat transfer between controllers. A pair of 10GbE embedded ports can be used to connect to a storage network: iSCSI or NAS protocol.
Each AIC HA401-LB2 platform controller has 6 PCIe 3.0 expansion slots. To maximize storage bandwidth, these PCIe slots should be equally distributed to organize three types of connections that involve the installation of appropriate adapters:
- 2 adapters for connecting storage systems to the storage network;
- 2 adapters for controller cache synchronization;
- 2 adapters for connecting external disk shelves.
AIC HA401-LB2 supports the installation of only Low-Profile PCIe adapters. For connecting external disk shelves in solutions based on RAIDIX, Broadcom SAS HBA adapters (LSI) are required. This manufacturer in the Low-Profile format produces only PCIe 3.0 x8 adapters.
The theoretical maximum bandwidth of a PCIe 3.0 x8 bus is 7.9 GB / s. Practice shows that the actual bandwidth of such an interface is no more than 6.5-7.5 GB / s. If this value is averaged up to 7GB / s, then the two adapters for connecting external disk shelves can be squeezed out to 14GB / s - this is the maximum possible RAIDIX bandwidth for this hardware platform.
With the above distribution of PCIe slots, from two to eight disk shelves can be directly connected to this platform:
- available two SAS HBA adapters per controller,
- 2 each ( SAS 9300-8E ) or 4 ( SAS 9305-16E ) SAS interface (mini-SAS HD) to the SAS HBA adapter, depending on the model.
Total from two to eight SAS interfaces per controller. Thus, taking into account the 24-disk AIC HA401-LB2 internal disk basket, the use of two 4U disk shelves on 60 HDD 3.5 "SAS / SATA allows you to organize a 144-disk storage system, the total height of the equipment in the rack is 12U. Connecting eight of these shelves makes it possible to build on this platform a storage system with a total capacity of 504 HDD 3.5 ", the total height of the equipment in the rack is 36U. When using disks on 10TB, the usable capacity of such storage will be up to 4.2 pb.
Further expansion and increase in throughput of RAIDIX storage is possible using hardware platforms with a large number of x8 PCIe slots (and / or PCIe x16 support) to install the required number of network and disk adapters.
Tasks that do not require high performance and storage capacity allow you to limit to a smaller number of adapters, you do not need to use all 6 PCIe slots, if there is no such need.
For example, if the project has enough internal disk basket AIC HA401-LB2, it will be enough to put one adapter per controller to synchronize the cache. If necessary, you can add 1-2 network adapters to the controller.
In the case of connecting one disk shelf for 60 HDD to get a 10GB / s bandwidth (for 2 controllers), you can get by with four PCIe slots per controller:
- 1 adapter to connect the storage system to the storage network (2 10GbE ports + 2 embedded 10GbE ports, 40GB / s in total or 5GB / s per controller);
- 2 adapters for controller cache synchronization;
- 1 adapter for connecting external disk shelves.
Thus, SDS RAIDIX allows you to:
- design the optimal storage for each specific task;
- pick up only the necessary components;
- provide for the possibility of expansion.
As a result, the end customer of the solution receives maximum design flexibility and no overpayment for unnecessary components.
RAIDIX Storage Configuration Examples
Dual HDD 3.5 ”RAIDIX dual-controller configuration
The maximum throughput of 3-4GB / s : 24 SAS 7200rpm disks, 150-200MB / s throughput from 1 disk, 2 RAID-6 or RAID-7.3 groups with 12 disks each.
Net Capacity : 182 TB for RAID-6 and 164 TB for RAID-7.3
Using two RAID groups of 12 disks allows you to bind them to separate controllers and remove them from each up to 1.8-2 GB / s. The storage network can be organized on iSCSI 10GbE, then to connect the storage system to the initiators you can use the 10GbE ports built into the server platform - 2 ports per controller.
If desired, you can make one large group of RAID-7.3 or RAID-6 on 24 disks, but only 1 controller will be active. Accordingly, the two integrated 10GbE ports may not be enough, you will have to install additional network interfaces for each controller: on a dual-port 10GbE card or FC adapters.
To connect the internal disk basket, you can use the 3008 SAS HBA adapter built into the platform (motherboard). To synchronize the cache with a margin of enough for one Broadcom SAS HBA-adapter SAS 9300-8e per controller.
24 HDD 3.5 ”10TB Specification on 10GbE Embedded Interfaces
| Component | Model | Quantity, pcs |
|---|---|---|
| Server platform | AIC HA401-LB2 | one |
| CPU | Intel Xeon E5-2620 V4 8core 2.1Ghz | 2 |
| Ram | Crucial by Micron DDR4 16GB | four |
| Bootable system media | Intel SSD DC S3500 Series (160GB, 2.5 '' SATA 6Gb / s) SSDSC2BB160G401 | four |
| SAS HBA Adapter | Broadcom SAS 9300-8e | 2 |
| Cache sync cables | mini-SAS HD (SFF-8644) to mini-SAS HD (SFF-8644) | 2 |
| HDD | HGST Ultrastar HE10 (3.5 '', 10TB, 256MB, 7200 RPM, SAS 12Gb / s) | 24 |
| RAIDIX license | On 26 disks, two-controller | one |
Dual-controller 84 HDD 3.5 ”RAIDIX configuration
Maximum throughput . To the server platform used in the previous specification, it is necessary to add an external disk shelf 60 HDD 3.5 ". We remove 3.5GB / s from the inner shelf, from the external 6.5GB / s. In total, you need to give and synchronize 10GB / s.
Net Capacity: 691.22 TB for RAID-6 and 654.84 TB for RAID-7.3
To connect the internal disk basket, we continue to use the 3008 SAS HBA adapter built into the platform (motherboard). To connect the external recycle bin and synchronize the cache, you will need three Broadcom SAS 9300-8e HBA adapters per controller: one for connecting the disks, two for the sync.
Connecting storage to the storage network can be organized in several ways, depending on the infrastructure and project requirements.
Option-1, iSCSI 10GbE . In order for the storage system to give out 10GB / s (or 80GB / s), you will need at least 4 10GbE ports per controller. Taking into account the 2 ports of 10GbE built into the platform, you will need to install one 2-port 10GbE adapter for each controller. However, in this case, when one controller drops, the second will be able to provide only half the bandwidth of the array - 40 Gb / s. Therefore, ideally, you need to install three 2-port 10GbE adapters per controller - 80 Gb / s from the controller.
Specification for 84 HDD 3.5 ”10TB, 10GbE external interfaces
| Component | Model | Quantity, pcs |
|---|---|---|
| Server platform | AIC HA401-LB2 | one |
| CPU | Intel Xeon E5-2637 v4 4core 3.5Ghz | four |
| Ram | Crucial by Micron DDR4 32GB | eight |
| Bootable system media | Intel SSD 240Gb S3520 SSDSC2BB240G701 Series | four |
| SAS HBA Adapter | Broadcom SAS 9300-8e | 6 |
| Cache sync cables | mini-SAS HD (SFF-8644) to mini-SAS HD (SFF-8644) | four |
| HDD for internal basket | HGST Ultrastar HE10 (3.5 '', 10TB, 256MB, 7200 RPM, SAS 12Gb / s) | 24 |
| Outer disk shelf | Disk shelf 60X10TB 4U60 G1 1ES0093 HGST | one |
| Adapters 10GbE | Intel Ethernet CNA X710 Series dual port 10GbE | 6 |
| RAIDIX license | On an unlimited number of drives, dual controller | one |
Option 2, FC 16Gbps . Compromise option for 4 ports 16GbFC per controller. We get 128GB / s for two controllers and 64GB / s for 1 controller failure (not 80GB / s, ideally, but not bad).
Specification for 84 HDD 3,5 ”10TB, 16GbFC external interfaces
| Component | Model | Quantity, pcs |
|---|---|---|
| Server platform | AIC HA401-LB2 | one |
| CPU | Intel Xeon E5-2637 v4 4core 3.5Ghz | four |
| Ram | Crucial by Micron DDR4 32GB | eight |
| Bootable system media | Intel SSD 240Gb S3520 SSDSC2BB240G701 Series | four |
| SAS HBA Adapter | Broadcom SAS 9300-8e | 6 |
| Cache sync cables | mini-SAS HD (SFF-8644) to mini-SAS HD (SFF-8644) | four |
| HDD for internal basket | HGST Ultrastar HE10 (3.5 '', 10TB, 256MB, 7200 RPM, SAS 12Gb / s) | 24 |
| Outer disk shelf | Disk shelf 60X10TB 4U60 G1 1ES0093 HGST | one |
| Adapters 16GbFC | QLE2672-CK - Fiber Channel 16Gb HBA dual port Qlogic QLE2672, PCIe 3.0 x8, 16/8/4, 2xSFP + SR | four |
| RAIDIX license | On an unlimited number of disks, dual-controller, with support for FC | one |
Dual-controller RAIDIX configuration on a 264 HDD 3.5 ”
The maximum bandwidth is 13-14 GB / s . Determined by the PCIe 3.0 x8 bandwidth (6.5-7 GB / s per slot).
Net Capacity: 2.183 PB for RAID-6 and 2.092 PB for RAID-7.3. We use the capacity of the internal disk basket of the platform and 4 external disk shelves 4U with 60 HDD 3.5 ".
To connect the internal disk basket, we continue to use the 3008 SAS HBA adapter built into the platform (motherboard). To connect external shelves, we use two Broadcom SAS 9305-16e HBA adapters per controller. To synchronize the cache, you will need two Broadcom SAS 9300-8e HBA adapters per controller. Total on controllers involved four PCIe-slots of the six.
Connecting storage to the storage network can be organized in several ways, depending on the infrastructure and project requirements.
Option-1, iSCSI 10GbE . On each controller, you can install 2 dual-port 10GbE adapters, taking into account the two built-in ports, we get six 10GbE interfaces per controller. The total bandwidth of the network connections of the platform will be 15GB / s, which covers the total bandwidth of the configuration (13-14GB / s). However, when a single controller falls, the system bandwidth will drop to 7.5 GB / s, since only half of the 10GbE ports will remain.
Specification for 264 HDD 3.5 ”10TB, 10GbE external interfaces
| Component | Model | Quantity, pcs |
|---|---|---|
| Server platform | AIC HA401-LB2 | one |
| CPU | Intel Xeon E5-2643 v4 6core 3.4Ghz | four |
| Ram | Crucial by Micron DDR4 32GB | sixteen |
| Bootable system media | HGST Ultrastar (2.5 '', 600GB, 128MB, 10,000 RPM, SAS 12Gb / s) HUC101860CS4204 | four |
| SAS HBA Adapter | Broadcom SAS 9300-8e | four |
| SAS HBA Adapter | Broadcom SAS 9305-16e | four |
| Cache sync cables | mini-SAS HD (SFF-8644) to mini-SAS HD (SFF-8644) | four |
| HDD for internal basket | HGST Ultrastar HE10 (3.5 '', 10TB, 256MB, 7200 RPM, SAS 12Gb / s) | 24 |
| Outer disk shelf | Disk shelf 60X10TB 4U60 G1 1ES0093 HGST | four |
| Adapters 10GbE | Intel Ethernet CNA X710 Series dual port 10GbE | four |
| RAIDIX license | On an unlimited number of drives, dual controller | one |
Option 2, FC 16Gbps . Install 4 ports 16GbFC per controller. The total bandwidth of the network connections of the platform will be 16GB / s, which covers the total bandwidth of the configuration (13-14GB / s). When a single controller falls, the system bandwidth drops to 8GB / s.
Specification for 264 HDD 3.5 ”10TB, 16GbFC external interfaces
| Component | Model | Quantity, pcs |
|---|---|---|
| Server platform | AIC HA401-LB2 | one |
| CPU | Intel Xeon E5-2643 v4 6core 3.4Ghz | four |
| Ram | Crucial by Micron DDR4 32GB | sixteen |
| Bootable system media | HGST Ultrastar (2.5 '', 600GB, 128MB, 10,000 RPM, SAS 12Gb / s) HUC101860CS4204 | four |
| SAS HBA Adapter | Broadcom SAS 9300-8e | four |
| SAS HBA Adapter | Broadcom SAS 9305-16e | four |
| Cache sync cables | mini-SAS HD (SFF-8644) to mini-SAS HD (SFF-8644) | four |
| HDD for internal basket | HGST Ultrastar HE10 (3.5 '', 10TB, 256MB, 7200 RPM, SAS 12Gb / s) | 24 |
| Outer disk shelf | Disk shelf 60X10TB 4U60 G1 1ES0093 HGST | four |
| Adapters 16GbFC | QLE2672-CK - Fiber Channel 16Gb HBA dual port Qlogic QLE2672, PCIe 3.0 x8, 16/8/4, 2xSFP + SR | four |
| RAIDIX license | On an unlimited number of disks, dual-controller, with support for FC | one |
Note
In the examples of configurations listed above for the control and heartbeat traffic, the embedded 1GbE ports are used.
The examples of RAIDIX configurations that are considered do not imply the use of spare disks to obtain maximum usable storage capacity.
RAIDIX supports allocation of spare-drives and provides switching to them in case of failures. The use of spare-disks and their number depends on the project conditions and is determined at the discretion of the customer, it is difficult to give general recommendations. In the absence of free slots for spare-disks, it is recommended to keep the required number of disks in the “cold” reserve for their prompt manual replacement.
Scope of application
Installing RAIDIX on the hardware platform options described above provides the following benefits:
- Maximum performance on sequential load, incl. multi-threaded . This is true for tasks where the key parameter is throughput - throughput (GB / s), and not the value of IOPS, which is important for random access. At the same time, high performance is guaranteed not only for a single load flow, but also for multiple parallel competing flows within a single storage.
- High resiliency (availability) . Support for dual-controller storage configuration, eliminating the presence of single points of failure. Support for RAID groups with robust coding to ensure data integrity and availability while simultaneously failing up to two (RAID-6) or more (RAID-7.3 and RAID-N + M) disks.
- Large usable capacity and high storage density . RAIDIX allows you to work effectively with large RAID groups with robust coding on 12-24 disks. At the same time HDD of large volume - 6-12TB can be used successfully.
- No loss in performance in case of disk failure, fast reconstruction . The problem with most storages is a dramatic drop in performance under load when a disk fails in a RAID group with checksums (RAID 5 or 6) when the array is in a degraded state. This can lead to service interruptions. Reconstruction of disk groups in such solutions takes a long time, the longer it takes, the greater the risk of failure of new disks and data loss. RAIDIX coding technology is implemented in such a way that storage performance is maintained even if the number of disks allowed by the configuration is lost. In this case, the reconstruction takes place several times faster than that of competitors.
- Ability to upgrade and vertical scaling . RAIDIX-based storage systems support granular expansion within the system by installing additional disks and disk shelves, adding HBA adapters, network interfaces, increasing processor power and RAM. The possibility of modernization involves the replacement of the above components with newer and more advanced in the event of their physical and moral obsolescence. Maximum flexibility based on the needs of the end customer.
This may be necessary in areas such as:
- CCTV
- media industry
- Backup
- HPC (supercomputers)
If the task does not require high bandwidth, while large storage capacity and density is required, you can limit the built-in 10GbE ports and somewhat reduce the requirements for the processor and RAM. This may be relevant for file and content storage archives. It is necessary to proceed from the conditions of a specific project and select the optimal solution.
Source: https://habr.com/ru/post/336086/
All Articles