324 Infiniband ports in one rack or Voltaire Grid Director 4700 switch
On the data center as a whole, we have almost all told and shown. Now we will gradually focus on interesting equipment installed there, or will be installed in the near future. The topic is, of course, very controversial, but from time to time such curious devices appear that it is simply impossible to resist the temptation. How do you, for example, have a 324-port QDR-Infiniband switch, with a total bandwidth of 50 Terabits? About him today and we will tell.
')
The switchboard is used in the cloud computing system of our partners, Oversan-Skalaksi, and is installed in the Oversan-Mercury data center. As with any other equipment of this class (and price), a careful study of the solutions available on the market preceded its selection and purchase. What was needed was an ultra-fast switch capable of taking over the organization of data transfer in the cluster file system GPFS and the transfer of ethernet traffic. This allows you to combine storage with disks and client machines into one network, providing, moreover, Internet access.
The data transfer bus must meet the requirements of cluster systems, that is, have a very high data transfer rate, with a margin of several years, and low latency. 40 Gigabit Ethernet is still a rather exotic and expensive bus, besides, the estimated cost of ownership (TCO) for a system of this size turned out to be completely space-based. As a result, the choice fell on Infiniband, as the technology tested, run-in, and relatively more affordable. And, of course, QDR-Infiniband can operate at speeds up to 40 Gbps, and the switch provides a delay of 100 to 300 nanoseconds when transmitting between ports.
The specialists of Oversun-Skalaxi conducted a whole study on what kind of switching matrix could meet all the imposed requirements. In the end, after long discussions, contacts and negotiations with vendors, Voltaire's Grid Director 4700 was chosen as the reference switch. About her and her products, few people know what makes the subject of discussion even more interesting. Meanwhile, the company was formed in 1997 and has Israeli-American roots. The initial benchmark was the production of data centers, high-load and cluster computing. Apparently, subsequently, cloud projects were added to this list. The company manufactures products, including powerful switching matrices, on Infiniband and Ethernet buses.
Here's what they say to themselves: “A line of switching matrices for servers and storage systems, as well as advanced management software, improve productivity in mission-critical applications, increase efficiency and reduce costs by consolidating infrastructure and lower power consumption.”
To the credit of Voltaire, their switches are used in systems that are included in the first positions with world ratings of supercomputers, and some Voltaire solutions are used in Bull, HP, IBM, NEC, Rackable, SGI and Sun systems.
You can write a separate book about how the switch was purchased, delivered and installed. We are not going to reveal these secrets yet, but rather we’ll dwell more directly on the hardware.
In the pictures, the switch is shown during installation so that you can take a picture of the "insides".
The four-thousandth series of Grid Director switches is designed for the cluster computing market, where lightning-fast speed, excellent controllability and scalable architecture are still required against the background of the ever-growing cluster size and complexity. The same is obviously required in the "clouds". In short, the Grid Director 4700 is a management switch containing 324 40 Gb / s Infiniband ports in a 19-unit chassis. The main switchboard modular board (backplane) provides a total bandwidth of 51.8 Terabits per second, while the delay when switching between ports is in the range from 100 to 300 nanoseconds. The numbers, they say, speak for themselves. For connection, special copper QSFP connectors and corresponding cables are used. In this case, the switch itself determines the characteristics of the connected cable and sets the optimal operating parameters. On the one hand, it seems to be a very useful function, but on the other hand, if you don’t want any surprises, buy Voltaire branded cables. And they, like, in general, and everything associated with high-speed data transfer, now is not cheap.
Through the HP DL-series with IB modules, the HP MSA disk storage with the GPFS cluster file system implemented on them is connected to the Grid Director 4700. This provides good disk I / O throughput in virtual instances. Actually the solution to the problem of input-output (I / O bottleneck) and Voltaire solutions are famous. Accordingly, using the Grid Director, Oversan-Skalaksi solves the main problem of all Russian hosts - low speed disk I / O. Also through Infiniband using its own design "Oversan-Skalaksi" Ethernet traffic is transmitted. And finally, the high bandwidth of Infiniband is needed to provide customers with a wide interconnect between virtual instances and the corresponding channel to the Internet through powerful uplinks to the Oversan-Mercury data center.
Shared Ethernet switching and Internet channeling is entrusted to Juniper EX82 switches. We also plan to tell about them in detail in the near future.
Up to 18 slots can be installed in the chassis, each of which contains a line card for QDR connectors. They, in turn, for 18 pieces on the board. From here also those 324 Infiniband-ports turn out. The same number of ports is achieved in another way - nine switching matrices of 36 ports each are installed in the chassis. All installed modules are managed using Unified Fabric Manager proprietary software.
Sockets for the matrix on the modular board.
The entire power supply is provided by six hot-swappable backup power supplies. This is done mainly for reliability, the consumption of the system itself is not so great - a maximum of 12 watts per port. At first glance it may seem a lot, but do not forget that the Grid Director 4700 alone takes almost half the rack.
Two fan modules, vertical and horizontal, are responsible for cooling. At the same time, the general direction of air movement remains standard, from the front - back. This is done so that the Grid Director can be installed in regular racks in standard data centers, although all ports at the switch are located behind, which is not so common.
And one more touch - when the modules are fully loaded, the switch weighs 170 kilograms. Solid, but not as much as it could have been - two engineers could easily cope with the installation.
Downstairs - cooling and power.
Part of the repository connects directly via SAS to servers with Infiniband modules.
Side view.
Indicators on the modules.
Nameplate close-up. Modest, but with taste.
')
The switchboard is used in the cloud computing system of our partners, Oversan-Skalaksi, and is installed in the Oversan-Mercury data center. As with any other equipment of this class (and price), a careful study of the solutions available on the market preceded its selection and purchase. What was needed was an ultra-fast switch capable of taking over the organization of data transfer in the cluster file system GPFS and the transfer of ethernet traffic. This allows you to combine storage with disks and client machines into one network, providing, moreover, Internet access.
The data transfer bus must meet the requirements of cluster systems, that is, have a very high data transfer rate, with a margin of several years, and low latency. 40 Gigabit Ethernet is still a rather exotic and expensive bus, besides, the estimated cost of ownership (TCO) for a system of this size turned out to be completely space-based. As a result, the choice fell on Infiniband, as the technology tested, run-in, and relatively more affordable. And, of course, QDR-Infiniband can operate at speeds up to 40 Gbps, and the switch provides a delay of 100 to 300 nanoseconds when transmitting between ports.
The specialists of Oversun-Skalaxi conducted a whole study on what kind of switching matrix could meet all the imposed requirements. In the end, after long discussions, contacts and negotiations with vendors, Voltaire's Grid Director 4700 was chosen as the reference switch. About her and her products, few people know what makes the subject of discussion even more interesting. Meanwhile, the company was formed in 1997 and has Israeli-American roots. The initial benchmark was the production of data centers, high-load and cluster computing. Apparently, subsequently, cloud projects were added to this list. The company manufactures products, including powerful switching matrices, on Infiniband and Ethernet buses.
Here's what they say to themselves: “A line of switching matrices for servers and storage systems, as well as advanced management software, improve productivity in mission-critical applications, increase efficiency and reduce costs by consolidating infrastructure and lower power consumption.”
To the credit of Voltaire, their switches are used in systems that are included in the first positions with world ratings of supercomputers, and some Voltaire solutions are used in Bull, HP, IBM, NEC, Rackable, SGI and Sun systems.
You can write a separate book about how the switch was purchased, delivered and installed. We are not going to reveal these secrets yet, but rather we’ll dwell more directly on the hardware.
In the pictures, the switch is shown during installation so that you can take a picture of the "insides".
The four-thousandth series of Grid Director switches is designed for the cluster computing market, where lightning-fast speed, excellent controllability and scalable architecture are still required against the background of the ever-growing cluster size and complexity. The same is obviously required in the "clouds". In short, the Grid Director 4700 is a management switch containing 324 40 Gb / s Infiniband ports in a 19-unit chassis. The main switchboard modular board (backplane) provides a total bandwidth of 51.8 Terabits per second, while the delay when switching between ports is in the range from 100 to 300 nanoseconds. The numbers, they say, speak for themselves. For connection, special copper QSFP connectors and corresponding cables are used. In this case, the switch itself determines the characteristics of the connected cable and sets the optimal operating parameters. On the one hand, it seems to be a very useful function, but on the other hand, if you don’t want any surprises, buy Voltaire branded cables. And they, like, in general, and everything associated with high-speed data transfer, now is not cheap.
Through the HP DL-series with IB modules, the HP MSA disk storage with the GPFS cluster file system implemented on them is connected to the Grid Director 4700. This provides good disk I / O throughput in virtual instances. Actually the solution to the problem of input-output (I / O bottleneck) and Voltaire solutions are famous. Accordingly, using the Grid Director, Oversan-Skalaksi solves the main problem of all Russian hosts - low speed disk I / O. Also through Infiniband using its own design "Oversan-Skalaksi" Ethernet traffic is transmitted. And finally, the high bandwidth of Infiniband is needed to provide customers with a wide interconnect between virtual instances and the corresponding channel to the Internet through powerful uplinks to the Oversan-Mercury data center.
Shared Ethernet switching and Internet channeling is entrusted to Juniper EX82 switches. We also plan to tell about them in detail in the near future.
Up to 18 slots can be installed in the chassis, each of which contains a line card for QDR connectors. They, in turn, for 18 pieces on the board. From here also those 324 Infiniband-ports turn out. The same number of ports is achieved in another way - nine switching matrices of 36 ports each are installed in the chassis. All installed modules are managed using Unified Fabric Manager proprietary software.
Sockets for the matrix on the modular board.
The entire power supply is provided by six hot-swappable backup power supplies. This is done mainly for reliability, the consumption of the system itself is not so great - a maximum of 12 watts per port. At first glance it may seem a lot, but do not forget that the Grid Director 4700 alone takes almost half the rack.
Two fan modules, vertical and horizontal, are responsible for cooling. At the same time, the general direction of air movement remains standard, from the front - back. This is done so that the Grid Director can be installed in regular racks in standard data centers, although all ports at the switch are located behind, which is not so common.
And one more touch - when the modules are fully loaded, the switch weighs 170 kilograms. Solid, but not as much as it could have been - two engineers could easily cope with the installation.
Downstairs - cooling and power.
Part of the repository connects directly via SAS to servers with Infiniband modules.
Side view.
Indicators on the modules.
Nameplate close-up. Modest, but with taste.
Source: https://habr.com/ru/post/94708/
All Articles