HPE Synergy Pro - Part IV. Our networks

Part I (Introduction)
Part II (Chassis and Server)
Part III - D3940 Disk Storage and SAS Switches

It is logical to divide this part into two pieces - network cards in servers and switches in the chassis. The cards have already been discussed in the second part, but it is interesting to discuss the details separately. In addition, it should be noted that the modules SY Composer, Image Streamer and FLM have their own backplane, separate from the general, and there is no information on this backplane.

Maps for servers do not say that a lot - 5 pieces, but in general - enough. Three cards - Eth + FCoE, two cards - pure Fiber Chanel. As I wrote earlier, just Ethernet cards as a class do not exist. All maps with PxE, that is, servers with them, can be loaded over the network. And they will be loaded, for this there Image Streamer'y.

• SY 2820C can - QLogic controller, two ports of 10 Gbit / s; not cheap, but already angry. Does not support remote direct memory access (RDMA) - this is not a frequent guest in IT infrastructures, although it is advanced - it allows you to directly address the memory of the processor of one server through the network to the processor of another server. It is very demanding of server resources: one card eats away 2 GB of server RAM, although the card itself is 1 GB, of which 900 MB is available for operations. Supports Jumbo frames - 9K. Since there are 4 tracks to the switches from each mezzanine slot, it is possible to squeeze 40 Gb / s Ethernet during the port aggregation;
  
  
• SY 3820C 10 / 20Gb CNA - a relative of the previous card; QLogic controller, two 10/20 Gbit / s ports (that is, a dozen symmetrical - 10 uplink, 10 downlink) - “pass, do not push”. Also requires 2 GB of RAM per card. With aggregation, you can already squeeze out 80 Gbit / s. The difference between the high and low cards in the GPL is about 200 USD.
  ')
  
• SY 3520C 10 / 20Gb CNA - here the controller from Emulex, in other characteristics, like the previous card. An interesting moment was in the HPE configurator, but it is no longer there, it is mentioned in the articles, but, for example, it cannot be found on the HPE website, as it is not in the HPE Product Bulletin, while Google or Yandex finds it on the same site without any problems HPE. But there are 4 topics about the problems arising from its use - maybe the dog is buried here?

Supported operating systems:

Microsoft:

• Microsoft Windows Server 2012 and 2012 R2;
• Microsoft Hyper-V Server 2012 and 2012 R2.

Linux:

• Red Hat Enterprise Linux x64 6.7, 6.8;
• Red Hat Enterprise Linux x64 7.2;
• SUSE Linux Enterprise Server 11SP4 x64;
• SUSE Linux Enterprise Server 12 & 12SP1 x64.

Virtualization:

VMware ESXi 5.5 U3, 6.0 U2 (here a little scary - vSphere 6.0 U2 is the minimum version of VMware for UEFI FCoE Boot from SAN support. Boot from SAN via VMware).

Also, all servers have 1 pc. 1 Gbit / s network card for iLO (remote control).
An interesting feature of all the cards is HPE Sea of ​​Sensors 3D support; Each blade server has 32 temperature sensors that send their data via a network card (and not via iLO) to the chassis control unit, which evaluates the temperature distribution in the volumetric model.

Below is a table of comparative characteristics, apparently a bit outdated, as the SY 620 Gen9 servers are also supported.


Storage Area Network Cards:

• SY 3530C 16 Gb FC - backward compatible with 8 Gb FC, capable of processing up to 600,000 IOPS on each port or 1.2 million on one if the second is disabled. Emulex controller.
• SY 3830C 16 Gb FC - all the same, only the controller from QLogic (apparently for those who do not like Emulex).

At a cost, the GPL card from Emulex is 400 bucks cheaper - this is not enough. In addition, the table below indicates that the warranty period for the cards is 1 year, not 3 years, I have not found any mention of this wonderful fact anywhere, so in the future it is worth having this moment in mind.


Go to the switches, in those recorded:

• HPE Virtual Connect SE 40 Gb F8 (master module, i.e. main);
• HPE SY 40 Gb F8 switch module;
• HPE Virtual Connect 20 Gb interconnect link module;
• HPE Virtual Connect 10 Gb interconnect link module;
• Brocade 16 Gb Fiber Channel SAN switch module;
• HPE SY 10 Gb pass-thru module.

The switch HPE Virtual Connect SE 40 Gb F8 is one of the key elements of Synergy, which contains a lot of new functionality (and in small print - competitor Cisco UCS Director). A pair of switches operates in active-passive mode.

Each pair of switches provides 16 downlinks for each server bay (14 Ethernet and 2 FC) if the servers are supplied with 10/20 Gb CNA cards. If delivered 2820, then options are possible - or 6 pcs. Ethernet, or 4 pcs. Ethernet plus 2 pcs. FC. With the help of Virtual Connect technology, these links can be further cut into FlexNIC 'and FlexHBA's.


The RJ-45 port, centered, is the most interesting, because it is signed as “for future use”. The two ports on the right (L1 and L2) and the two ports on the left (L3 and L4) are the communication ports between the switch and the Interconnect Link modules (see below). G7 and G8 ports are the clustering ports between the switches in the chassis. The remaining ports (G1 - G6) are 40 Gb / s QSFP + uplinks.

It is not necessary to buy such a switch in each chassis, up to three (or five, but more on that below) the chassis can be deployed using a pair of VC SE 40 Gb F8, and then installing HPE 10/20 Gb ILM modules (Interconnect Link Module).

You can read about HPE Virtual Connect technology here - HP Virtual Connect FlexFabric Cookbook (313 pages) is a very voluminous topic, and the purpose of this article is not to disclose it. Or, if you are interested in the basics, here you can download Virtual Connect for Dummies for free - checked before publishing - I was sent =)

Next - HPE Synergy 40 Gb Switch Module.


The main difference between the modules of the HPE Virtual Connect SE 40Gb F8 and the HPE Synergy 40 Gb Switch Module is, in fact, support by the first module of Virtual Connect technologies and the presence of FC support in the first licensed function. Of the interesting things - support by Super Jumbo frames - 12 Kb, which do not support CNA yet - we wait and wait.


The HPE Synergy Interconnect Link 10G And 20 G Modules are optional modules in an additional chassis.


They are connected to the two left and / or two right ports on the HPE Virtual Connect SE 40Gb F8. What is special: the module with one “hole” (10G) allows you to connect up to 4 additional chassis to the SY 12000 head chassis (i.e. just one entity will have 5 pieces of SY 1200 Frame), and with two “20G” holes "(I.e., only one entity will have 3 pieces of SY 1200 Frame) - up to 2 additional chassis, but at a higher speed. In the case of connecting to the main chassis, an additional chassis is a feature - the HPE Virtual Connect SE 40Gb F8 switches are not installed as a pair in the head chassis, but 1 pc. between the main (first) and additional (second) module. The connection example from 5 chassis is shown in the picture:



What else can be noted? The architecture is non-blocking, without oversubscription; the table below says that RTT (round trip time latency) is less than 8 nanoseconds, more recent documentation (July 29, 2016) already writes about “less than 5 nanoseconds”. In order for such a miracle to work, connect the modules with special cables: or copper - direct connection cable (DAC - Direct Attach Copper Cable) with a maximum length of 2.1 meters (i.e. only inside the rack) or active optical cables, maximum length - up to 15 meters. According to the GPL price tag, the longest optical cable is about 6 times more expensive than the longest copper cable.


The storage network switch is non-blocking, currently only available for use. The switch is manufactured by the respected Brocade company, there are 48 ports with 4, 8 or 16 Gbit / s FC auto-detection in the switch (by the way, you can plug 8 Gbit / s SFP + modules into ports 25-31), some ports are aggregated. 12 ports for server compartments, 8 ports for external, and 4 groups of QSFP uplinks (Quad Small Form-factor Pluggable).


There is an option POD (ports on demand, licensing of open ports as the need arises) - the tricky feature of extracting money from the user - “pay for the hardware, and then also for it to fully work.” That is, you can buy with 12 “open” licensed ports (in any combination of external and internal ports), and if you need more, then buy a key for another 12 ports or PowerPack (licenses for additional functionality). The two left and two right port groups allow you to install QSFP - 4x16 Gbit / s FC. These ports can be used as individual 16 Gb / s ports, as 64 Gb / s FC ISL, or as a 128 Gb / s trunk to connect to the SAN director. For virtual environments, supports NPIV.

The minimum FabricOS version with which the switch is delivered is FOS 7.2.0a.

What can be connected to.

16 Gb / s FC switches and SAN-director:

• SN8000B Director Switch (Enterprise);
• SN6500B Fiber Channel SAN Switch;
• Brocade 16Gb SAN Switch for HPE BladeSystem c-Class;
• SN6000B Fiber Channel Switch (Midmarket);
• SN3000B Fiber Channel Switch (Entry);
• Brocade 16 Gb SAN Switch for HPE c-Class BladeSystem;
• Brocade 16Gb Fiber Channel SAN Switch Module for HPE Synergy (to the same switches in other chassis).

8 Gbps FC Switches:

• 8/24 SAN Switch;
• 8/8 SAN Switch;
• Brocade 8 Gb SAN Switch for HPE c-Class BladeSystem.

Virtual Connect Switches:

• HPE Virtual Connect 8Gb 24-port Fiber Channel Module;
• HPE Virtual Connect 8Gb 20-port Fiber Channel Module;
• HPE Virtual Connect 16 Gb 24-port Fiber Channel Module;
• Virtual Connect FlexFabric 10Gb / 24-Port Module for c-Class Blade System;
• HPE Virtual Connect FlexFabric-20/40 F8 Module for c-lass Blade System.

Multiprotocol routers and gateways:

• Multi-protocol Extension Blade;
• SN4000B FC SAN Extension Switch;
• 1606 SAN Extension Switch.

Below is a table with possible options for purchasing a switch:


And finally - a favorite security module. Transmits data from 12 bays with blade servers, auto detection of speed, maximum - 10 Gbit / s. There are no brains, autodetect, diagnose, monitor and everything else through Composer.


The table of correspondence of connection ports and compartments where PTM is installed (PTM - Pass-thru Module, CM - Compute Module):


To refresh your memory:


From some nuances it is possible to note the originality of support - separately for the module it is not, only as part of the SY 12000 chassis.

The description of the “Calculation-Storage-Transfer of Information” block can be considered complete, the next article will be devoted to Composer and Image Streamer. Separately, I want to say hello to the Russian marketing of the manufacturer - Composer - this is translated as “composer”, not “composer”, but it’s scary to speak about the translation of Image Streamer.