Network Evolution - New 25/50/100 GbE Standard
Modern Ethernet networks have long become an integral part of our lives, and business processes without them are completely unthinkable. And their capacity is that characteristic, the growth of which is extremely in demand. The speed of 1 Gbit / s has long become so commonplace that even providers provide private individuals. But as soon as it comes to transmitting any serious information flows, this is clearly not enough, and it can be said that the universal standard is now 10G, and with connectors not Base-T, but SFP +, which can be used as copper patch cords at distances up to 7 meters ( rack or group of racks), and optical communication lines. Well, for high-loaded data lines (aggregation levels, cluster interconnect for high-performance computing, connecting powerful storage systems), 40G QSFP + is used, which is the combination of four 10G SFP + lines.
But server performance is growing, data flows, too, the load on the network is getting higher, and therefore there is a long overdue need to switch to high speeds. For a long time, there are 100G products in the formats of CFP, CFP2, CFP4, but they are inadequately expensive for use in the server infrastructure and it was necessary to create an affordable product. For this, the 25G Consortium was created, which, together with the Ethernet Alliance, was engaged in the development of a new industrial standard designed to raise speeds in typical networks by 2.5 times. Where the development of a standard ends, the 2550100 Alliance , created to promote products to the masses, comes into play . We will talk now about the results of their work and the degree of readiness of the standard for a massive invasion of the market.
So, comes to us a new standard. The base speed in it will be 25 Gbit / s, the combination of two lines gives 50 Gbit / s, and four - 100 Gbit / s.
')
To begin with, the new standard is an evolutionary, not a revolutionary development. From the point of view of work with the network, all the schemes of work, protocols, technologies and recommendations remain the same, only the physical layer changes. All the foundations for this have been laid for a long time: the standard is described in IEEE Std 802.3ba-2010, and since 2011 the corresponding equipment has been produced, albeit in the form of trunk routers, corresponding cards and transceivers. 4 years ago it was the peak of technology and they cost a lot of money, but now we are talking about creating commonly used equipment for the data center in the form of low-power multi-port rack switches, network adapters and corresponding cables. In terms of connectors, we have to replace SFP + with SFP28 and, accordingly, QSFP + with QSFP28.
It is curious that the SFP28 and QSFP28 connectors differ little from the already familiar SFP + and QSFP +, which compares favorably with the CFP solutions used in the backbone hardware. The situation with power consumption is also similar - the new generation lasers provide less than 3.5 W of energy release per 100G port, which allows creating compact solutions suitable for mass use.
From the point of view of backward compatibility, everything is in order - all the emerging equipment currently supports work at speeds of 10 / 40G, therefore migration to the new standard can be carried out gradually, adding new 25 / 100G switches to the old 10 / 40G equipment and smoothly pulling the speed to the new standard on the required sections of the network.
As for the readiness of the new standard for mass production and implementation, here, in our opinion, everything is pretty good. Already there is a full range of necessary equipment implemented in iron, PCB and silicon, for example:
- switches
- network cards
transceivers
- cables
- equipment for testing lines
Accordingly, everything is in order and wide support of the standard from the side of component manufacturers: such products as Qlogic, Avago, Cavium, Finisar, Broadcom, Ixia, Mellanox have already presented their products. A general list of companies that supported the new standard is so great that it is easier to give a link than to give here the entire list. True, we can not mischievously notice that if we have already joined this list (about our product a little later), then such a small company like Intel will not.
In addition to jokes, this is somewhat difficult to believe, but it seems that Intel completely missed the appearance of 25 / 100G, concentrating on fine-tuning its X710 solution. Well, competitors obviously do not slow down a piece of the market by massively releasing their network adapters.
And it will also be rather curious to look at the speed with which the leading manufacturers of network equipment will start to switch to a new standard and what arguments will highlight their decision on merhant silicon against the background of competitors - in the near future everything will be built around the same matrix Broadcom Tomahawk .
Another important fact that promises to work hard on the popularity of the new standard is that its cost will not be radically different from the current 10 / 40G solutions. Usually innovations (or innovations, for those who are so accustomed) for quite a long time scare away disproportionately high prices from themselves, as a result of which only those who can no longer live without switching to the most productive technologies become the pioneers of their introduction. At the same time, the emphasis is planned on the mass character of the solution, therefore, even taking into account the “mark-up for novelty” (and it is impossible to get rid of it completely), new solutions in price-performance ratio promise to surpass their predecessors at the start.
Who needs network infrastructure with such high bandwidth? Well, let's look at obvious implementation examples.
Use in Web-Scale installations
If we consider typical large-scale installations in the form of racks completely clogged by servers (40U, 80 nodes by 1 / 2U), then 2-port 10G adapters at each node are replaced with similar 25G adapters. Accordingly, at the TOR-level, 4 10G switches, for example Eos420 (48 ports 10G, 6 ports 40G) can be replaced by 2 100G switches, in which out of 32 100G ports 20 use break-out cables to connect to 25G nodes and the remaining 12 work on uplink.
Advantages of this solution:
Use in High Performance Computing
The HPC currently uses the 56G InfiniBand most often. Accordingly, it can be “head-on” replaced with 100G connections within a block of 32 servers.
Advantages:
Use to connect storage
Already, block or file access to the storage via Ethernet turns out to be more practical than FiberChannel and has closely matched the performance of InfiniBand. The new interface will further exacerbate this situation, in effect making all other interfaces unnecessary.
Advantages:
And finally, about our participation in 2550100.
For our part, we presented the Eos 720 switch with 32 QSFP28 ports. This model was a continuation of our existing family of Eos switches, and was created within the framework of our beloved ideology BareMetal Switch, that is, it has the ONIE installation environment that supports the installation of various network operating systems. The switchboard is already fully prepared and will soon be available in our laboratory, and as regards the support from the operating system, we are now closely cooperating with their manufacturers on compatibility issues.
The main characteristics of the switch are:
So, summing up, we can say that the new standard is already fully ready to enter the market and within a year we should expect the mass appearance of commercial products. So plan your infrastructure now.
But server performance is growing, data flows, too, the load on the network is getting higher, and therefore there is a long overdue need to switch to high speeds. For a long time, there are 100G products in the formats of CFP, CFP2, CFP4, but they are inadequately expensive for use in the server infrastructure and it was necessary to create an affordable product. For this, the 25G Consortium was created, which, together with the Ethernet Alliance, was engaged in the development of a new industrial standard designed to raise speeds in typical networks by 2.5 times. Where the development of a standard ends, the 2550100 Alliance , created to promote products to the masses, comes into play . We will talk now about the results of their work and the degree of readiness of the standard for a massive invasion of the market.
So, comes to us a new standard. The base speed in it will be 25 Gbit / s, the combination of two lines gives 50 Gbit / s, and four - 100 Gbit / s.
')
To begin with, the new standard is an evolutionary, not a revolutionary development. From the point of view of work with the network, all the schemes of work, protocols, technologies and recommendations remain the same, only the physical layer changes. All the foundations for this have been laid for a long time: the standard is described in IEEE Std 802.3ba-2010, and since 2011 the corresponding equipment has been produced, albeit in the form of trunk routers, corresponding cards and transceivers. 4 years ago it was the peak of technology and they cost a lot of money, but now we are talking about creating commonly used equipment for the data center in the form of low-power multi-port rack switches, network adapters and corresponding cables. In terms of connectors, we have to replace SFP + with SFP28 and, accordingly, QSFP + with QSFP28.
It is curious that the SFP28 and QSFP28 connectors differ little from the already familiar SFP + and QSFP +, which compares favorably with the CFP solutions used in the backbone hardware. The situation with power consumption is also similar - the new generation lasers provide less than 3.5 W of energy release per 100G port, which allows creating compact solutions suitable for mass use.
From the point of view of backward compatibility, everything is in order - all the emerging equipment currently supports work at speeds of 10 / 40G, therefore migration to the new standard can be carried out gradually, adding new 25 / 100G switches to the old 10 / 40G equipment and smoothly pulling the speed to the new standard on the required sections of the network.
As for the readiness of the new standard for mass production and implementation, here, in our opinion, everything is pretty good. Already there is a full range of necessary equipment implemented in iron, PCB and silicon, for example:
- switches
- network cards
transceivers
- cables
- equipment for testing lines
Accordingly, everything is in order and wide support of the standard from the side of component manufacturers: such products as Qlogic, Avago, Cavium, Finisar, Broadcom, Ixia, Mellanox have already presented their products. A general list of companies that supported the new standard is so great that it is easier to give a link than to give here the entire list. True, we can not mischievously notice that if we have already joined this list (about our product a little later), then such a small company like Intel will not.
In addition to jokes, this is somewhat difficult to believe, but it seems that Intel completely missed the appearance of 25 / 100G, concentrating on fine-tuning its X710 solution. Well, competitors obviously do not slow down a piece of the market by massively releasing their network adapters.
And it will also be rather curious to look at the speed with which the leading manufacturers of network equipment will start to switch to a new standard and what arguments will highlight their decision on merhant silicon against the background of competitors - in the near future everything will be built around the same matrix Broadcom Tomahawk .
Another important fact that promises to work hard on the popularity of the new standard is that its cost will not be radically different from the current 10 / 40G solutions. Usually innovations (or innovations, for those who are so accustomed) for quite a long time scare away disproportionately high prices from themselves, as a result of which only those who can no longer live without switching to the most productive technologies become the pioneers of their introduction. At the same time, the emphasis is planned on the mass character of the solution, therefore, even taking into account the “mark-up for novelty” (and it is impossible to get rid of it completely), new solutions in price-performance ratio promise to surpass their predecessors at the start.
Who needs network infrastructure with such high bandwidth? Well, let's look at obvious implementation examples.
Use in Web-Scale installations
If we consider typical large-scale installations in the form of racks completely clogged by servers (40U, 80 nodes by 1 / 2U), then 2-port 10G adapters at each node are replaced with similar 25G adapters. Accordingly, at the TOR-level, 4 10G switches, for example Eos420 (48 ports 10G, 6 ports 40G) can be replaced by 2 100G switches, in which out of 32 100G ports 20 use break-out cables to connect to 25G nodes and the remaining 12 work on uplink.
Advantages of this solution:
- 150% higher throughput
- Requires 2 times less network infrastructure (switches, cables) in the rack -> below CAPEX
- 2 times less than TOR switches -> below OPEX
- All existing virtualization and overlay network technologies are supported (VXLAN, NVGRE, SPB etc)
Use in High Performance Computing
The HPC currently uses the 56G InfiniBand most often. Accordingly, it can be “head-on” replaced with 100G connections within a block of 32 servers.
Advantages:
- 78% higher throughput
- Below CAPEX
- Simple system scaling
- Unified network for all types of traffic
Use to connect storage
Already, block or file access to the storage via Ethernet turns out to be more practical than FiberChannel and has closely matched the performance of InfiniBand. The new interface will further exacerbate this situation, in effect making all other interfaces unnecessary.
Advantages:
- Unified network infrastructure for servers and storage -> below CAPEX and OPEX
- Highest data transfer rate required for all-flash arrays and large storage systems
- Low latency with RDMA and support for protocols used in storage systems
- Easy scaling
- Convenient management
- Ability to select the method of data storage (file, block or object) while maintaining a single interface
And finally, about our participation in 2550100.
For our part, we presented the Eos 720 switch with 32 QSFP28 ports. This model was a continuation of our existing family of Eos switches, and was created within the framework of our beloved ideology BareMetal Switch, that is, it has the ONIE installation environment that supports the installation of various network operating systems. The switchboard is already fully prepared and will soon be available in our laboratory, and as regards the support from the operating system, we are now closely cooperating with their manufacturers on compatibility issues.
The main characteristics of the switch are:
- 32 ports 100G QSFP28 / 128 25GbE SFP28 using breakout cables
- Broadcom Tomahawk BCM56960 switching matrix, 3.2Tbps, 64-byte packet forwarding rate: 2400 Mpps
- Intel Atom 2538 processor
- MAC Address Table - 136K Unified Forwarding Table (UFT)
- delay less than 500 ns (PHY-less)
- VXLAN / NVGRE hardware processing
- Optimization for data centers: 802.1Qau, 802.1Qaz, 802.1Qbb, DCBX, EVB (802.1Qbg), MLAG, 32-way ECMP
- BMS platform with ONIE: supported by Cumulus Linux (after update), Broadcom ICOS (in November)
- SDN switching support: OpenFlow 1.0, 1.2, 1.3, Open API
- Broadview toolkit support for full network control
- Failsafe power scheme 1 + 1
So, summing up, we can say that the new standard is already fully ready to enter the market and within a year we should expect the mass appearance of commercial products. So plan your infrastructure now.
Source: https://habr.com/ru/post/259147/
All Articles