Huawei HyperMetro - tested and implemented

The main features of the third generation DSS Huawei Oceanstor at Habré have already been told , but there is little information about the access function to that “stretched” between sites. We will try to close this omission. This material was prepared as a result of the implementation of one of our customers in the banking environment.

Detailed description

The main purpose of HyperMetro is protection against data center failure, and only then storage system, disk shelf, etc. The feature became available with the V300R003C00SPC100 firmware in December 2015.

In fact, this is the provision of one-time read and write access to a logical volume that has two independent copies on the storage system, which are located on different technological sites. The topology of the HyperMetro solution is shown below.

Figure 1. HyperMetro Topology

For arbitration in case of failures, Quorum is used, which is installed on the third site, has access to both arrays via Ethernet and is an application for Linux (SUSE / RHEL / Ubuntu). Reading and writing to a distributed volume is possible from any of the sites to any of the two storage systems by exchanging locks between arrays by using a data change log (DCL), which protects against the possibility of writing from different hosts to one block. It should also be noted that the cache entry of both storage systems runs in parallel, rather than sequentially. The following is the writing process in HyperMetro.
')
Figure 2. The order of operations when writing to a distributed volume

MPIO Huawei Ultrapath (bundled with Oceanstor V3) is responsible for prioritizing local read / write to avoid cross-site traffic. Performance balancing between sites is allowed. It is possible to use not only FC, but also iSCSI - both for access to hosts and for replication.

After creating the HyperMetro relationship, both volumes are considered symmetrical and equivalent. However, in the event of a quorum failure and communication between the storage systems, a priority side is set for each of the HyperMetro relationships in order to avoid the “split brain” situation. It is possible to use consistency groups to merge several pairs of volumes. In case of failure of one of the copies in the overwhelming majority of cases, there is no loss of access to the data.

In the absence of a third site, HyperMetro can be used in a static mode (Static Priority Mode). This mode provides for automatic decision-making in case of failures in favor of storage systems with a high priority (preferred site). In case of failure of the main storage, the non-preferred array will have to start access manually, changing the priority of the HyperMetro relationship for each volume pair or consistency group. Also, the arrays go into unqualified mode if the quorum or all access paths to it fail. If you place a quorum on one of the sites, then if it falls, access to data on the other site will not be provided - manual switching will be required.

Pros:

• HyperMetro is licensed for each storage system, not for mirrored volume;
• in connection with the general third generation firmware code in the third generation Oceanstor storage system, this feature is available from 2600v3 to the 18800v3 Hi-End array;
• works not only on FC, but also on IP between two main sites;
• Replication, tiered storage, virtualization, snapshots, and other storage features on HyperMetro-protected volumes are available.

Minuses:

• requires a separate Ethernet card to connect to the arbiter (you cannot use dedicated ports to control storage for these purposes);
• Requires installing the MPIO Ultrapath driver on the hosts.

Competitive decisions

Due to the fact that the volume distributed between sites is technologically based on normal synchronous replication, then most of the major manufacturers already have similar solutions: HPE 3Par Peer Persistence, IBM HyperSwap, HDS Global Active Device, Fujitsu Storage Cluster, Dell Compellent Live Volume. And apart, you can highlight data protection solutions between sites, which appeared much earlier than the above and are based on unique solutions: IBM SVC Stretched Cluster, EMC VPLEX Metro, and Netapp MetroCluster.

In this material we did not pursue the goal to compare products from different manufacturers. However, in the comments we will try to answer your questions regarding the comparison of any specific points in the architecture of solutions.

Implementation

Since HyperMetro was a new and untested function, we decided to first test together with the customer on the basis of the equipment from the demofound. The testing used two customer server platforms connected via DWDM. United SAN network based on two Brocade factories. Each site has one Huawei OceanStor 5500 V3 storage system with SAS 10k rpm disks and SSD, x86 blade server farm with VMware.

On two storage systems, LUNs are consistently created, consistency groups, and the HyperMetro relationship is configured. After this, volumes from both storage systems are addressed to the required hosts. Test virtual machines running IOmeter were running on the servers. When simulating the failure of one of the storage systems, a significant drop in performance (> 10%) was not observed. When loading CPU controllers less than 50%.

After a full setup, even hard switching off the array (pulling out the power cables from the controllers) did not lead to loss of access. After the connection between the arrays is restored, the volumes are synchronized manually (you can configure the automatic mode in the settings of each HyperMetro relationship - Recovery Policy). If necessary, the synchronization process can be paused, deployed, or restarted to fully resynchronize volumes. It should be noted that access switching via another storage system does not occur if the array is manually rebooted via the graphic menu or via the command interface.

In the process, we worked out a simulation of various failures (see table below). The behavior of the system is fully consistent with that described in this table and in the documentation of the manufacturer.

#	Equipment operability			Connection efficiency			Result
	SHD №1	Storage number 2	Quorum	№1 <-> Q	№2 <-> Q	№1 <-> №2
one.	+	+	+	+	+	+	Everything works as usual
2	-	+	+	+	+	+	Access to this goes through a working storage system.
3	+	-	+	+	+	+
four.	- *	+	+	-	+	-
five.	+	- *	+ *	+	-	-
6	+	+	-	-	-	+	Access to that goes through both storage systems. They are for 20 sec. go to unqualified priority mode
7	+	+	-	+	+	+
8.1.	+	- *	-	+	+	+	Access to this goes through the Priority storage system. If Priority broke, then manual activation of access with Non-Priority storage is necessary.
8.2.	+	- *	-	+	+	+	Access to this is stopped, you need to manually enable access to the storage system
9.	+	+	+	-	+	+	Access to that goes through both storage systems.
	+	+	+	+	-	+
eleven.	+	+	+	+	+	-	Arbitration occurs. Access to this goes through the Priority storage system.
12.	+	+	+	-	+	-	Access to that goes through the storage system number 2.
13.	-	+	+	+	-	+	Access stopped. Manual activation of access with storage No. 2 is required

* 4 Fall of the site with storage without quorum
* 5 Fall of the storage area with quorum
* 8.1 A storage breakdown occurred more than 20 seconds after item 7
* 8.2 A storage breakdown occurred less than 20 seconds after step 7

After successful testing, the customer has approved the target configuration. Implementation of the solution after such testing was carried out promptly and without problems. The system works like a clock for 5 months already. Technical support in 24 * 7 mode is provided by our Service Center (“Jet” is the highest-level service partner of Huawei - Certified Service Partner 5-stars). Without false modesty, we note that the customer is completely satisfied with the capabilities of HyperMetro and the performance of the solution based on OceanStor. He plans to further increase the amount of protected data.

The material was prepared by Dmitry Kostryukov, Senior Storage Design Engineer, Jet Infosystems.