The nuances of ESXi, FlexFabric, 10 Gbit and NFS bundles
In this article, I would like to present useful information gathered during the design and implementation of a fault-tolerant virtualization environment. Particular attention is paid to the nuances of HP Virtual Connect FlexFabric and the configuration of the VMware vSphere 5.5 hypervisor when using Ethernet 10 Gbit and NFS as a Datastore.
Network diagram
"Iron"
Blade-cart "HP BladeSystem c7000" with a pair of modules "HP VC FlexFabric 10/24".
HP BL460c Gen8 servers with HP FlexFabric 10Gb 536FLB network card.
Network switches "Cisco Nexus 5k".
Storage system "NetApp FAS8020".
')
Channel aggregation is one of the main means of achieving a high resiliency of the virtual environment, so you need to use this at all levels of traffic flow. In our case:
To get the maximum benefit from 10 Gbit, it is recommended to enable Jumbo Frames at all levels:
The MTU value in HP Virtual Connect is “wired” to 9216 and cannot be changed .
How to install Jumbo Frames on ESXi .
Unfortunately, CDP is not supported by the “HP VC FlexFabric” modules, so there is no sense to include its support on the hypervisor.
Excerpt from the documentation: “ Virtual Connect does not support CDP. VC does support the industry standard protocol called Link Layer Discovery Protocol (LLDP) by default. LLDP is functionally equivalent to CDP, although the two protocols are not compatible. "
In the matter of using the “Flow Control” mechanism, we decided to adhere to the recommendation of NetApp and disable it at all “levels” (ESXi, FlexFabric, Network, Storage).
NetApp recommendation: “ Modern network equipment and protocols handle port congestion better than those in the past. NFS and iSCSI as implemented in ESXi use TCP. TCP has built-in congestion management, making Ethernet flow control unnecessary. Furthermore, it will send you a remote frame. Although it has been recommended that it has been approved by NetApp for network access control. "
In the configuration of the “HP VC FlexFabric” modules, Flow Control is enabled by default only on downlink (value “Auto”), and on uplink — off is turned off.
"ON" - all ports control (if autoneg) or flowcontrol turned on (non-autoneg).
"OFF" - all ports will support the flow control (if autoneg) or flowcontrol turned off (non-autoneg).
“Auto” - all uplink / stacking links will be “OFF”, and all server links will be “ON”.
The command to turn off: #set advanced-networking FlowControl = off
Interesting articles on this topic:
Virtual Connect FlexFabric interconnect modules and Ethernet Flow Control
NETAPP vs VMWARE FLOW CONTROL DILEMMA
Configuring Flow Control on VMware ESXi and VMware ESX
Smart Link mode must be enabled for all vNet (Ethernet Network) configurations in FlexFabric. This is necessary for the hypervisor to properly run the balancing in the virtual switch.
Excerpt from the documentation: “ HP's Virtual Connect supports a feature called smart link, if you’re uplink ports lose link. This is a feature that is available on the HP GbE2, GbE2c and most ProCurve switches. I called there Link State Tracking. "
It is recommended to separate virtual machine traffic from management traffic (vMotion, Management, NFS, FT). To increase the reliability of the virtual environment, we used a standard switch for management traffic, rather than distributed, although it has several advantages (for example, LACP support).
In this configuration, virtual switches recommend using load balancing mode "Route based on the originating virtual port id".
The “Route Based on IP Hash” mode ( NetApp recommendation ) cannot be used, since it requires combining its uplink (virtual switch) to a trunk via the 802.3ad protocol, and HP VC FlexFabric does not provide such an opportunity for downlink to servers.
The rest of the load balancing policy settings are:
Network failure detection: Link status only.
Notify switches: Yes.
Failback: Yes.
For each service (vMotion, Management, NFS, FT) it is recommended to create a separate VMkernel Port. Vmk for NFS traffic (Available Services remains empty) must be created on the same subnet as the NFS exports. In our case:
For vMotion vmkernel adapters, HP recommends setting “failover order” mode in Active / Standby:
“To ensure that it’s not a problem.” This will ensure that all virtual connect module. ”
It is recommended to change the default values ​​of some parameters for working with NFS exports. For each vSphere host in Advanced Settings, set the following values:
NFS.HeartbeatFrequency = 12
NFS.HeartbeatTimeout = 5
NFS.HeartbeatMaxFailures = 10
Net.TcpipHeapSize = 32
Net.TcpipHeapMax = 512
NFS.MaxVolumes = 256
NFS.MaxQueueDepth = 64
Recommendations are described in the following documents:
Best Practices for running VMware vSphere on Network Attached Storage
NFS mounts on an ESXi / ESX host
VMware vSphere 5 on NetApp Clustered Data ONTAP
Best Practices for Running VMware vSphere on Network-Attached Storage (NAS)
HP Virtual Connect FlexFabric Cookbook
FC Cookbook for HP Virtual Connect
HP Virtual Connect for the Cisco Network Administrator
HP Virtual Connect Manager Command Line Interface
HP HP BladeSystem Virtual Connect Forum
Network diagram
"Iron"
Blade-cart "HP BladeSystem c7000" with a pair of modules "HP VC FlexFabric 10/24".
HP BL460c Gen8 servers with HP FlexFabric 10Gb 536FLB network card.
Network switches "Cisco Nexus 5k".
Storage system "NetApp FAS8020".
')
Channel aggregation
Channel aggregation is one of the main means of achieving a high resiliency of the virtual environment, so you need to use this at all levels of traffic flow. In our case:
| ESXi | Flexfabric | Network | Storage |
| vSphere vSwitch in “Port ID” mode | Shared Uplink Set + Auto Mode (LACP) | EtherChannel (LACP) | lif |
Jumbo frames
To get the maximum benefit from 10 Gbit, it is recommended to enable Jumbo Frames at all levels:
| VMkernel port | vSwitch | Flexfabric | Network | Storage | |
| Value | 9000 | 9000 | 9216 | 9216 | 9000 |
The MTU value in HP Virtual Connect is “wired” to 9216 and cannot be changed .
How to install Jumbo Frames on ESXi .
Cisco Discovery Protocol
Unfortunately, CDP is not supported by the “HP VC FlexFabric” modules, so there is no sense to include its support on the hypervisor.
Excerpt from the documentation: “ Virtual Connect does not support CDP. VC does support the industry standard protocol called Link Layer Discovery Protocol (LLDP) by default. LLDP is functionally equivalent to CDP, although the two protocols are not compatible. "
Flow control
In the matter of using the “Flow Control” mechanism, we decided to adhere to the recommendation of NetApp and disable it at all “levels” (ESXi, FlexFabric, Network, Storage).
NetApp recommendation: “ Modern network equipment and protocols handle port congestion better than those in the past. NFS and iSCSI as implemented in ESXi use TCP. TCP has built-in congestion management, making Ethernet flow control unnecessary. Furthermore, it will send you a remote frame. Although it has been recommended that it has been approved by NetApp for network access control. "
In the configuration of the “HP VC FlexFabric” modules, Flow Control is enabled by default only on downlink (value “Auto”), and on uplink — off is turned off.
"ON" - all ports control (if autoneg) or flowcontrol turned on (non-autoneg).
"OFF" - all ports will support the flow control (if autoneg) or flowcontrol turned off (non-autoneg).
“Auto” - all uplink / stacking links will be “OFF”, and all server links will be “ON”.
The command to turn off: #set advanced-networking FlowControl = off
Interesting articles on this topic:
Virtual Connect FlexFabric interconnect modules and Ethernet Flow Control
NETAPP vs VMWARE FLOW CONTROL DILEMMA
Configuring Flow Control on VMware ESXi and VMware ESX
Smart link
Smart Link mode must be enabled for all vNet (Ethernet Network) configurations in FlexFabric. This is necessary for the hypervisor to properly run the balancing in the virtual switch.
Excerpt from the documentation: “ HP's Virtual Connect supports a feature called smart link, if you’re uplink ports lose link. This is a feature that is available on the HP GbE2, GbE2c and most ProCurve switches. I called there Link State Tracking. "
Virtual switches
It is recommended to separate virtual machine traffic from management traffic (vMotion, Management, NFS, FT). To increase the reliability of the virtual environment, we used a standard switch for management traffic, rather than distributed, although it has several advantages (for example, LACP support).
vSphere vSwitch Load Balancing
In this configuration, virtual switches recommend using load balancing mode "Route based on the originating virtual port id".
The “Route Based on IP Hash” mode ( NetApp recommendation ) cannot be used, since it requires combining its uplink (virtual switch) to a trunk via the 802.3ad protocol, and HP VC FlexFabric does not provide such an opportunity for downlink to servers.
The rest of the load balancing policy settings are:
Network failure detection: Link status only.
Notify switches: Yes.
Failback: Yes.
VMkernel Port
For each service (vMotion, Management, NFS, FT) it is recommended to create a separate VMkernel Port. Vmk for NFS traffic (Available Services remains empty) must be created on the same subnet as the NFS exports. In our case:
| VMkernel port | Available Services | Network label | VLAN ID | MTU |
| vmk0 | vmotion | vmotion | one | 9000 |
| vmk1 | Management | Management | 2 | 9000 |
| vmk2 | - | Nfs | 3 | 9000 |
For vMotion vmkernel adapters, HP recommends setting “failover order” mode in Active / Standby:
“To ensure that it’s not a problem.” This will ensure that all virtual connect module. ”
NFS Advanced Settings
It is recommended to change the default values ​​of some parameters for working with NFS exports. For each vSphere host in Advanced Settings, set the following values:
NFS.HeartbeatFrequency = 12
NFS.HeartbeatTimeout = 5
NFS.HeartbeatMaxFailures = 10
Net.TcpipHeapSize = 32
Net.TcpipHeapMax = 512
NFS.MaxVolumes = 256
NFS.MaxQueueDepth = 64
Recommendations are described in the following documents:
Best Practices for running VMware vSphere on Network Attached Storage
NFS mounts on an ESXi / ESX host
Other nuances
- The blade server network card must be compatible with Virtual Connect modules. Compatibility can be checked in QuickSpecs from HP.
- It is advisable to update the firmware of the Virtual Connect modules to the latest available version, but it should be done very carefully when checking the compatibility of the FW blade servers and the recycle bin.
- Included with the Virtual Connect modules, SFP transceivers do not come, so plan ahead for the physical switching circuit, and buy the right transceivers.
- Virtual Connect allows you to guarantee and set bandwidth limits for subnets (at the level of vNet / Ethernet Network / VLAN). This should be used, for example, to limit VLANs with ESXi Management traffic up to 1 Gbit and VLAN guarantees for NFS from 4 Gbit to 10 Gbit.
Literature
VMware vSphere 5 on NetApp Clustered Data ONTAP
Best Practices for Running VMware vSphere on Network-Attached Storage (NAS)
HP Virtual Connect FlexFabric Cookbook
FC Cookbook for HP Virtual Connect
HP Virtual Connect for the Cisco Network Administrator
HP Virtual Connect Manager Command Line Interface
HP HP BladeSystem Virtual Connect Forum
Source: https://habr.com/ru/post/268859/
All Articles