Clusterize NetApp storage systems using handy switches
Clustering storage systems is now gaining momentum, especially in the light of rapidly developing Flash technologies that require more controllers capable of handling high-performance drives. In addition to supported cluster switches, there are many others that can be temporarily used for these purposes. In this article, I would like to give an example of setting up several switches that I have tested for a cluster network. I am sure that many other switches will also work, because this is normal Ethernet
Switch Connection Wiring for Cluster Interconnect
When you have on hand (at least temporarily) more than two FAS controllers, a cluster switch may be needed in the case of:
')
Both for temporary cluster configurations consisting of more than two controllers, and for permanent ones, a supported cluster switch is required. At the moment there are only two such switches, NetApp CN1610 and Cisco Nexus 5596. If you bought four or more controllers at once, then you certainly have this switch. And if this is a temporary configuration from the examples above, most likely there is no such switch. And what to do? The answer is obvious: configure the switch that you now have at hand to create a temporary configuration.
Despite the fact that other switches are not officially supported, they certainly work:
The following switches were tested with MTU 1500 :
With Cisco Nexus 5548UP MTU 9000 :
The following cDOT 8.3RC1, 8.3.1RC1 operating systems with repositories were tested:
If necessary, we reconfigure the MTU for NetApp cluster ports:
In production, NetApp does not recommend using Cluster Interconnect switches for anything other than cluster connectivity of NetApp storages; you need separate switches for this, although it is technically possible to use the same switches for both tasks.
Let me remind you that starting with cDOT 8.3, the new SLM feature works by default, and in the case of online migration it is necessary, on nodes that accept migrating moon, to allow hosts to be told that the moon is available in new, additional ways.
Make sure that the host multipassing driver has discovered these new paths.
In the case of a migration of volyum, which is used for NFS / CIFS (SMB) file protocols, after the migration, do not forget to move the LIF to the node where the volyum has moved.
More details on how to get nodes out of a cluster .
More on how to set up zoning for cluster storage .
Comments on errors and suggestions for changes in the text please send to the LAN .
Switch Connection Wiring for Cluster Interconnect
When you have on hand (at least temporarily) more than two FAS controllers, a cluster switch may be needed in the case of:
- Converting a FAS 2240 / 255x into a shelf and then connecting it to a new FAS system (which also has a shelf with disks), which requires online transfer without stopping access to the data. Both systems should work cDOT OS
- Upgrade 7M to cDOT with subsequent online data migration back . Learn more about the benefits of Cluster-Mode .
- Updating controller firmware (although it can be done without stopping, using only two controllers, but if you have another pair of controllers with disks, why not install them)
- In case you need to perform a more rational reboot of disks (for example, to get the ADP feature: Root-Data Partitioning or Storage Pool )
- And etc.
')
Both for temporary cluster configurations consisting of more than two controllers, and for permanent ones, a supported cluster switch is required. At the moment there are only two such switches, NetApp CN1610 and Cisco Nexus 5596. If you bought four or more controllers at once, then you certainly have this switch. And if this is a temporary configuration from the examples above, most likely there is no such switch. And what to do? The answer is obvious: configure the switch that you now have at hand to create a temporary configuration.
Despite the fact that other switches are not officially supported, they certainly work:
The following switches were tested with MTU 1500 :
- HP Pro Curve 6120XG (Image stamp Z.14.44, 1564) from HP c7000 with 10Gb links to 2240 and 8020
- Cisco WS-C3850-48T (IOS 03.03.03SE RELEASE SOFTWARE (fc2)) with 1Gb links to 2240 and 8020
HP Pro Curve Settings 6120XG from HP c7000
# HP Pro Curve 6120XG from HP c7000 swbl2# sh ver Image stamp: /ws/swbuildm/Z_zinfandel_fip_t4b_qaoff/code/build/vern(Z_zinfandel_fip_t4b_qaoff) Aug 26 2013 16:32:58 Z.14.44 1564 Boot Image: Primary swbl2# sh vlan Status and Counters - VLAN Information Maximum VLANs to support : 256 Primary VLAN : TECH Management VLAN : VLAN ID Name | Status Voice Jumbo ------- -------------------- + ---------- ----- ----- 1 DEFAULT_VLAN | Port-based No No 210 NetApp-Cluster | Port-based No No vlan 210 name "NetApp-Cluster" untagged 19-20,22 tagged Trk1-Trk2 # Trk1-Trk2 - connection of both HP switches to each other and to Cisco WS-C3850-48T no ip address exit wbl1# sh vlans 210 Status and Counters - VLAN Information - VLAN 210 VLAN ID : 210 Name : NetApp-Cluster Status : Port-based Voice : No Jumbo : Yes Port Information Mode Unknown VLAN Status ---------------- -------- ------------ ---------- 22 Untagged Learn Up swbl2# sh cdp neighbors 22 detail CDP neighbors information for port 22 Port : 22 Device ID : clA-02 Address Type : IP Address : 169.254.166.38 Platform : NetApp Release 8.3RC1: Fri Oct 31 20:13:33 PDT 2014FAS8020 Capability : Device Port : e0b Version : NetApp Release 8.3RC1: Fri Oct 31 20:13:33 PDT 2014FAS8020 Configuring Cisco WS-C3850
#Cisco WS-C3850-48T interface Vlan210 description NetApp-Cluster no ip address interface GigabitEthernet2/0/10 switchport access vlan 210 switchport mode access flowcontrol receive on spanning-tree portfast ! core1-co2#sh cdp neighbors Capability Codes: R - Router, T - Trans Bridge, B - Source Route Bridge S - Switch, H - Host, I - IGMP, r - Repeater, P - Phone, D - Remote, C - CVTA, M - Two-port Mac Relay Device ID Local Intrfce Holdtme Capability Platform Port ID netapp2 Gig 1/0/10 149 H FAS2240-2 e0b netapp2 Gig 1/0/9 149 H FAS2240-2 e0a netapp2 Gig 2/0/9 149 H FAS2240-2 e0d netapp2 Gig 2/0/10 149 H FAS2240-2 e0c netapp2 Gig 1/0/40 149 H FAS2240-2 e0M netapp1 Gig 2/0/8 166 H FAS2240-2 e0d netapp1 Gig 2/0/7 166 H FAS2240-2 e0c netapp1 Gig 1/0/8 166 H FAS2240-2 e0a netapp1 Gig 1/0/7 166 H FAS2240-2 e0b netapp1 Gig 1/0/39 166 H FAS2240-2 e0M clA-02 Gig 2/0/12 142 H FAS8020 e0f clA-02 Gig 2/0/6 142 H FAS8020 e0e clA-01 Gig 2/0/11 168 H FAS8020 e0f clA-01 Gig 1/0/6 168 H FAS8020 e0e core1-co2#sh vlan VLAN Name Status Ports ---- -------------------------------- --------- ------------------------------- 1 default active Te1/1/3, Te1/1/4, Gi2/0/37, Gi2/0/38, Gi2/0/41, Gi2/0/48, Te2/1/3 Te2/1/4 210 NetApp-Cluster active Gi1/0/6, Gi1/0/7, Gi1/0/8, Gi1/0/9, Gi1/0/10, Gi2/0/6, Gi2/0/7 Gi2/0/8, Gi2/0/9, Gi2/0/10, Gi2/0/11, Gi2/0/12 VLAN Type SAID MTU Parent RingNo BridgeNo Stp BrdgMode Trans1 Trans2 ---- ----- ---------- ----- ------ ------ -------- ---- -------- ------ ------ 210 enet 100210 1500 - - - - - 0 0 ore1-co2#sh ver Cisco IOS Software, IOS-XE Software, Catalyst L3 Switch Software (CAT3K_CAA-UNIVERSALK9-M), Version 03.03.03SE RELEASE SOFTWARE (fc2) Technical Support: http://www.cisco.com/techsupport Copyright (c) 1986-2014 by Cisco Systems, Inc. Compiled Sun 27-Apr-14 18:33 by prod_rel_team ROM: IOS-XE ROMMON BOOTLDR: CAT3K_CAA Boot Loader (CAT3K_CAA-HBOOT-M) Version 1.2, RELEASE SOFTWARE (P) Ipbase License Type: Permanent Next reload license Level: Ipbase cisco WS-C3850-48T (MIPS) processor with 4194304K bytes of physical memory. Processor board ID FOC1801U0WU 11 Virtual Ethernet interfaces 104 Gigabit Ethernet interfaces 8 Ten Gigabit Ethernet interfaces 2048K bytes of non-volatile configuration memory. 4194304K bytes of physical memory. 250456K bytes of Crash Files at crashinfo:. 250456K bytes of Crash Files at crashinfo-2:. 1609272K bytes of Flash at flash:. 1609272K bytes of Flash at flash-2:. 0K bytes of Dummy USB Flash at usbflash0:. 0K bytes of Dummy USB Flash at usbflash0-2:. 0K bytes of at webui:. Base Ethernet MAC Address : 50:1c:bf:6c:65:00 Motherboard Assembly Number : 73-14444-06 Motherboard Serial Number : FOC180112WY Model Revision Number : M0 Motherboard Revision Number : A0 Model Number : WS-C3850-48T System Serial Number : FOC1801U0WU Switch Ports Model SW Version SW Image Mode ------ ----- ----- ---------- ---------- ---- * 1 56 WS-C3850-48T 03.03.03SE cat3k_caa-universalk9 INSTALL 2 56 WS-C3850-48T 03.03.03SE cat3k_caa-universalk9 INSTALL With Cisco Nexus 5548UP MTU 9000 :
Nexus 5548UP Software version 7.1 (1) N1 (1)
banner motd # Nexus 5548 NetApp Reference Configuration File (RCF) version 1.3-48p (2012-10-15) # feature lacp cdp enable cdp advertise v1 cdp timer 5 snmp-server community cshm1! ro errdisable recovery interval 30 errdisable recovery cause pause-rate-limit policy-map type network-qos cluster class type network-qos class-default mtu 9216 system qos service-policy type network-qos cluster spanning-tree port type edge default port-channel load-balance ethernet source-dest-port interface port-channel1 switchport mode trunk spanning-tree port type network interface Ethernet1/1 description Cluster Node 1 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/2 description Cluster Node 2 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/3 description Cluster Node 3 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/4 description Cluster Node 4 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/5 description Cluster Node 5 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/6 description Cluster Node 6 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/7 description Cluster Node 7 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/8 description Cluster Node 8 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/9 description Cluster Node 9 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/10 description Cluster Node 10 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/11 description Cluster Node 11 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/12 description Cluster Node 12 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/13 description Cluster Node 13 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/14 description Cluster Node 14 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/15 description Cluster Node 15 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/16 description Cluster Node 16 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/17 description Cluster Node 17 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/18 description Cluster Node 18 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/19 description Cluster Node 19 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/20 description Cluster Node 20 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/21 description Cluster Node 21 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/22 description Cluster Node 22 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/23 description Cluster Node 23 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/24 description Cluster Node 24 no lldp transmit no lldp receive spanning-tree port type edge spanning-tree bpduguard enable interface Ethernet1/25 description Inter-Cluster Switch ISL Port 25 (port channel) no lldp transmit no lldp receive switchport mode trunk channel-group 1 mode active interface Ethernet1/26 description Inter-Cluster Switch ISL Port 26 (port channel) no lldp transmit no lldp receive switchport mode trunk channel-group 1 mode active interface Ethernet1/27 description Inter-Cluster Switch ISL Port 27 (port channel) no lldp transmit no lldp receive switchport mode trunk channel-group 1 mode active interface Ethernet1/28 description Inter-Cluster Switch ISL Port 28 (port channel) no lldp transmit no lldp receive switchport mode trunk channel-group 1 mode active interface Ethernet1/29 description Inter-Cluster Switch ISL Port 29 (port channel) no lldp transmit no lldp receive switchport mode trunk channel-group 1 mode active interface Ethernet1/30 description Inter-Cluster Switch ISL Port 30 (port channel) no lldp transmit no lldp receive switchport mode trunk channel-group 1 mode active interface Ethernet1/31 description Inter-Cluster Switch ISL Port 31 (port channel) no lldp transmit no lldp receive switchport mode trunk channel-group 1 mode active interface Ethernet1/32 description Inter-Cluster Switch ISL Port 32 (port channel) no lldp transmit no lldp receive switchport mode trunk channel-group 1 mode active end The following cDOT 8.3RC1, 8.3.1RC1 operating systems with repositories were tested:
- FAS 2240
- FAS 8020
If necessary, we reconfigure the MTU for NetApp cluster ports:
network port broadcast-domain modify -broadcast-domain Cluster -mtu 1500-or-9000 -ipspace Cluster If necessary, convert SwitchLess-Cluster to Switched:
cluster1::*> network options switchless-cluster modify -enabled true If necessary, we migrate cluster LIFs
If a node has only one clustered LIF, create second paths and migrate them one by one, so that all nodes see each other at least in one way: through swichless switching or via switches.
Do not migrate all the LIFs of a single node at once, a fallen off node with HA turned on will immediately reboot!
HA will certainly work, but why, if you can do without it?
Do not migrate all the LIFs of a single node at once, a fallen off node with HA turned on will immediately reboot!
HA will certainly work, but why, if you can do without it?
cluster1::> network interface show -role cluster cluster1::> network interface modify -vserver vsm0 -lif clus1_1 -home-node node01 -home-port e0c -auto-revert false cluster1::> network interface modify -vserver vsm0 -lif clus2_1 -home-node node02 -home-port e0c -auto-revert false # LIF # Cluster Node Management , LIF. cluster1-01::> network interface revert -vserver vsm0 -lif clus1_1 cluster1-02::> network interface revert -vserver vsm0 -lif clus2_1 Be sure to test the clustered network from the advanced mode:
cluster1::*> cluster ping-cluster -node local Host is node1 Getting addresses from network interface table... Local = 10.254.231.102 10.254.91.42 Remote = 10.254.42.25 10.254.16.228 Ping status: .... Basic connectivity succeeds on 4 path(s) Basic connectivity fails on 0 path(s) ................ Detected 1500 byte MTU on 4 path(s): Local 10.254.231.102 to Remote 10.254.16.228 Local 10.254.231.102 to Remote 10.254.42.25 Local 10.254.91.42 to Remote 10.254.16.228 Local 10.254.91.42 to Remote 10.254.42.25 Larger than PMTU communication succeeds on 4 path(s) RPC status: 2 paths up, 0 paths down (tcp check) 2 paths up, 0 paths down (udp check) In production, NetApp does not recommend using Cluster Interconnect switches for anything other than cluster connectivity of NetApp storages; you need separate switches for this, although it is technically possible to use the same switches for both tasks.
Block access:
Let me remind you that starting with cDOT 8.3, the new SLM feature works by default, and in the case of online migration it is necessary, on nodes that accept migrating moon, to allow hosts to be told that the moon is available in new, additional ways.
cluster1::> lun mapping add-reporting-nodes -vserver vserver_name -path lun_path -igroup igroup_name [-destination-aggregate aggregate_name | -destination-volume volume_name] Make sure that the host multipassing driver has discovered these new paths.
NAS:
In the case of a migration of volyum, which is used for NFS / CIFS (SMB) file protocols, after the migration, do not forget to move the LIF to the node where the volyum has moved.
cluster1::> network interface migrate -vserver vs0 -lif datalif1 -source-node vs0 -dest-node node2 -dest-port e0c More details on how to get nodes out of a cluster .
More on how to set up zoning for cluster storage .
Comments on errors and suggestions for changes in the text please send to the LAN .
Source: https://habr.com/ru/post/265763/
All Articles