About InfiniBand: how we reduced ping from 7 μs to 2.4 μs (and test results)

^{InfiniBand switch SX6005. 12 FDR 56Gb / s ports per unit, switching 1.3Tb / s.}

Many people believe that InfiniBand is “space”. That is, it is believed that it is expensive and necessary only for “supercomputers” (HPC) with a capacity of 1-2 Petaflops and with huge volumes of processed data. However, using this technology, you can organize not only the fastest interconnects in clusters, but also drastically reduce delays in critical applications. Specifically - to do something that can be solved using Ethernet , but more economical and faster. Here is an example.

Task

One of our major customers from the financial sector had a problem with the speed of the two applications. The specifics of the application was that it was necessary to process a large number of transactions with minimal delay. 6-7 μs latency - this is the best results that they achieved by upgrading servers and maximum software refinement. Further possible optimizations promised improvements of 0.3-0.5 µs. We came and said that we could reduce the delays in half.
')

Decision

We suggested making communication between servers using InfiniBand. Of course, the customer’s specialists had heard about this technology, but they needed specific tests on the site to check everything personally. To do this, we prepared a set of demo equipment, and they allocated several servers for it. "Battle" server did not want to touch for obvious reasons. So, we built a fragment of a live network using IB connections at the customer’s site.

To solve this problem, we took the equipment manufactured by Mellanox. The main criteria for choosing this particular manufacturer was that the Mellanox switches have “universal ports”, which are programmatically defined either as an InfiniBand port or as regular Ethernet — which will then allow seamless integration into the customer's existing Ethernet network. In addition, Mellanox produces a full range of equipment (including switches, network cards for any servers, interface cables, and so on), which in the future will make it possible to assemble the entire solution regardless of the type and manufacturer of the servers available to the customer.

Customers installed their applications on servers and rolled tests (it seems, using real tasks of almost the previous financial day) and testing began.

Test results

First Test Architecture:

^{IB equipment: ConnectX-3 HCA, SX6025 FDR switch, HP Proliant DL380 G7 server, RHEL OS 6.4 MRG 2.2.}

Testing steps:

Run IB Read latency test:
On the 1st node: ib_read_lat -a
On the 2nd node: ib_read_lat –a <node1_ip>

Run IB Write latency test:
On the 1st node: ib_write_lat -a -R
On the 2nd node: ib_write_lat –a -R <node1_ip>

Run the IB Send latency test:
On the 1st node: ib_send_lat -a -R
On the 2nd node: ib_send_lat –a -R <node1_ip>

The following test (IPoIB performance):

1. Run the Sockperf UDP test:
On the 1st node: sockperf sr -i
2- : sockperf pp -t 10 –i

2. Sockperf TCP:
1- : sockperf sr -i --tcp
2- : sockperf pp -t 10 –i --tcp

sockperf sr -i
2- : sockperf pp -t 10 –i

2. Sockperf TCP:
1- : sockperf sr -i --tcp
2- : sockperf pp -t 10 –i --tcp



Second test results

Test Sockperf UDP:
For a packet of 16 bytes: avg-lat = 9.133 (std-dev = 1.226)
For a 64 byte packet: avg-lat = 9.298 (std-dev = 1.268)
For a packet of 256 bytes: avg-lat = 9.605 (std-dev = 1.031)
For a packet of 1024 bytes: avg-lat = 10.791 (std-dev = 1.066)
For a packet of 4096 bytes: avg-lat = 17.107 (std-dev = 1.548)
For packet 16384 bytes: avg-lat = 34.512 (std-dev = 2.098)
For a packet of 65506 bytes: avg-lat = 96.502 (std-dev = 3.181)

Test Sockperf TCP:

For a packet of 16 bytes: avg-lat = 10.509 (std-dev = 1.185)
For a 64 byte packet: avg-lat = 10.506 (std-dev = 1.154)
For a packet of 256 bytes: avg-lat = 11.552 (std-dev = 1.128)
For a packet of 1024 bytes: avg-lat = 12.409 (std-dev = 1.168)
For a packet of 4096 bytes: avg-lat = 18.991 (std-dev = 1.506)
For packet 16384 bytes: avg-lat = 32.937 (std-dev = 1.952)
For a packet of 65506 bytes: avg-lat = 76.926 (std-dev = 3.066)

The final result is 2.36 - 2.37 μs. The best results are about 2 µs. It was possible to achieve more than the customer required.

And even after the tests, I saw the very feeling of deep moral satisfaction experienced by people who know that now their system will work more efficiently. Without replacing servers.

Ethernet and InfiniBand

I must say: one thing is not a replacement for another. It's like comparing a passenger car with a tank: one is convenient and fast, and on the second one you can go to Europe. It is important that from IB you should not expect a large number of services and the possibility of building a complex routing scheme - this is an absolute lot of Ethernet. The IB architecture is “flat”.

In this case, as a result of simplicity - InfiniBand is very easy to upgrade and scale. Our practice of deploying local networks after installing the equipment is to go through and plug in the contacts, and then sit for an hour at the terminal, and from this hour, 40 minutes will go to check the ready-made solution. In the case of Ethernet rebuilding, architecture changes are required where there are so many steps — and because of the flexibility and some kind of vindictiveness of the technology in relation to those who do not know all the subtleties, the procedure is similar to signing a contract with a bunch of items in small print. And there are some chances that complex architecture will bring surprises in the future.

Rebuilding an Ethernet network with parameters like IB will take 3-4 times longer on average. The speed of solving problems of restoring the IB network is a maximum of 4 hours at any time of the day. This indicator can be easily registered in the SLA - we have never been faced with the fact that even in difficult cases, time at least somehow crushed. Of course, you can register the same time for Ethernet - but then an entire team of system administrators must go to the site.

Well, the main difference is speed.
Ethernet, like FibreChannel, is a "limited" technology. Of course, you can provide 100Gb bandwidth on Ethernet by aggregating an unlimited number of 10Gb links. But! First is the limit. Secondly - what will be the cost of this solution? And if you add the cost of operation? We still need to understand how much space, electricity and the cost of heat removal in the data center will require a 100 Gb core-level switch, but they need more than one. At IB - 100 Gb / s is not the limit, and the cost of this solution (both CAPEX and OPEX) will be several times less, since there is less space, less loss, fewer cables, less electricity and air conditioning costs, less man hours. on operation and so on. In general, IB is sick simply faster and cheaper for tasks of this class.

Need a practical example?
You are welcome. Our cloudy data center at some point demanded high speeds. At the same time, the architecture should be very flexible, quickly and painlessly modernized, reconfigured for specific individual client requirements (this is one of the CRIC’s cloud trumps). Having calculated the rate of change, we certainly chose IB.

Manufacturers

There used to be four main manufacturers - Qlogic, Mellanox, Voltaire and Topspin. Mellanox bought Voltaire about 3 years ago. Topspin became part of Cisco in 2005, and Intel acquired Qlogic in 2012 (now the equipment is mainly OEP). As a result, there is now a profile player - Mellanox, providing the most complete range. Its OEM is HP, IBM, Violin and so on.

Perspectives

Among the “space-based” high-speed solutions, InfiniBand turns out to be very economical both in implementation costs and in service. Only 2-3 years ago such solutions were needed only for very serious tasks with industry significance, but now there are more and more enterprise-class projects. Everything is very simple: the amount of data increases, the load from applications grows, server performance grows and, as a result, other data exchange speeds between servers and application speeds are required.

Customers say this: “We are not losing anything, we are not changing the technology, we are not throwing out Ethernet, but we want to complement our InfiniBand infrastructure.”

Why come to us with such questions?

Many customers are very pleased that, firstly, you can look at the equipment live (we are constantly working on our projects and in our data centers), do tests on our sites (as I gave the example above), introduce a pilot system and then everything to introduce. We also have people who can optimize software for specific large-scale tasks taking into account all the features of networks - this is the next round of work with highload. And we try to tell as much as possible in a professional manner about all the possible technological options for solving the problem, without imposing anything on anyone.

UPD. Judging by the comments, it is not entirely clear why such a decision was chosen. Summarizing: the customer had to solve the problem of reducing delays from 7 μs to 3 μs on specific applications. It was possible to solve this by installing low-latency Ethernet equipment, which the customer was able to calculate himself. At the same time, the technical specialists of the customer decided to compare IB with their version.

We conducted tests at the customer’s site, and its specialists confirmed the following:

• IB fully solves the problem and even exceeds customer expectations.
• The solution is very easy to deploy on combat servers with continuous services.
• The solution is convenient to operate.
• The cost of implementation and maintenance costs are much lower than for Ethernet solutions.

In this case, the customer used IB-compatible modules, which determined the choice of vendor.

PS If you are interested in this topic, then on September 12 I will conduct a webinar, connect . If you need to immediately calculate something or understand whether it is possible to conduct tests directly on your site, write to AFrolov@croc.ru.