InterSystems Database Mirroring. Creating and testing mirrors. Part 2

In the previous part, the configuration of the mirror was considered - the high availability technology of the InterSystems Database Mirroring DBMS Caché.
In this article scenarios of breaks (refusals) and the reaction of the mirror to them will be considered.

Breaks can be both planned and unplanned.
Scheduled - this is when we need to stop the server to update the operating system, database version, version of the application system and we control the process.

Examples of scheduled breaks:
')

• reboot the OS, for example, to install updates;
• reboot the DBMS;
• hardware update server.

Unscheduled - when something happens that makes it impossible for the server to work with the clients of the information system, and without our knowledge. The reason for this may be:

• DBMS hang;
• operating system crash;
• emergency (Reset, Power Off);
• failure of server hardware;
• failure of network equipment;
• war, epidemic, blizzard, space black holes.

Failover

Automatic overcoming of a failure for a Caché mirror occurs only in case of a failure / interruption on the Primary-server, and in such article such cases will be considered. Of course, there may be scenarios when a problem occurs on the Backup server, but in this case the IP continues to work, and there is no physical mirroring until the administrator manually restores the Backup server. Another scenario is possible when both servers failed at once - but in this case, a break in the IS is inevitable, and its settlement is performed “manually”.

Basic scenario for automatic failover:

1. The Primary-server serves the clients of the IC that work with one of the mirrored databases, and the Backup-server is “ready” (stand by);
2. A failure occurs on the Primary server;
3. The backup server makes sure that the Primary server is down, automatically becomes the Primary server and continues to serve the clients of the information system;
4. The former Primary-server is restored (for example, by the administrator), becomes a new Backup-server and enters the “stand by” mode for the new Primary-server.

What can go wrong?
Let us imagine a situation when the network interface on the Primary-server fails, but not for long, but it is enough that the Backup-server has already managed to “decide” to become the Primary-server. The result is a split brain situation.

Split brain

Split brain - brain cancer is a state of complete desynchronization of cluster servers, when both servers are working and each of them considers that it is Primary at the moment.
Such a situation may arise, for example, if the Backup-server “mistakenly” decided that the Primary-server is in “Down”, as a result, two Primary-servers appear at once in the cluster, which try to simultaneously serve the users' requests - a cluster collapse.
For users of IP, this situation is expressed in an indefinite interruption of work, which is restored by the administrator “manually”.
The fastest way to restore such a situation is to re-create the cluster by restoring data from a backup or from an asynchronous “mirror” .

AgentContactRequiredForFailover

The decisive factor for the behavior of a Caché mirror during a failure / interruption is the value of the AgentContactRequiredForFailover = yes / no parameter.

AgentContactRequiredForFailover = yes

In the case of AgentContactRequiredForFailover (ACRF) = yes (by default), with failover, the ISC Agent on both servers must be in contact at the time of the interruption. Such a situation is possible for cases of unscheduled Cache failure, planned Cache reload, planned OS reboot. Therefore, in this mode, for all other cases, automatic failover will not occur!

AgentContactRequiredForFailover = no

This mode of mirroring allows for failover even without online contact between server agents. This mode may be needed to cover situations like Power Off primary server, network adapter failure, or the network itself between servers, etc., when there is no connection with the Primary server. In this mode, the decision to automatically switch the Backup-server to the Primary-server state is determined by the user function in the ZMIRROR program located in the area (% SYS).

Test results

Of course, when building a failover solution, we expect automatic overcoming of failure for most situations, otherwise why all this. Currently, the Caché mirror is able to automatically process almost all possible scenarios.
We tested how the mirror behaves for several typical scenarios at different ACRF values.
The latest version of Caché 2012.2 RC has been tested. Yes - means automatic failure override for this scenario.

N	Scenarios	ACRF = Yes	ACRF = No
one	Planned or emergency reboot, stop Caché DBMS on Primary	Yes	Yes
2	OS scheduled reboot	Yes	Yes
3	Emergency stop OS, Reset, Power Off	Not	Yes
four	Disable network interface on Primary	Not	Yes
five	Power Off at the same time Primary and Backup, and then download first Primary	Yes	Yes
6	Power Off at the same time Primary and Backup, and then downloading first Backup	Not	Not

As a result of testing, it turned out that an unpleasant scenario is the disconnection of the network interface (No. 4). Therefore, it is recommended to install a backup network channel between the mirror servers
Server configuration example:

In case of simultaneous failure of both servers - scenarios 5-6 (long interruption in power supply, for example), it is important to observe the commissioning sequence - the Primary-server should be activated first. Otherwise, the Backup-server becomes Primary, and after the start of the former Primary-server there are two of them - as a result of the split brain.

As can be seen from the table, the Caché synchronous mirror with ACRF = yes (the default value) provides only those abnormal situations when the network between the cluster servers is “alive”.
For other situations, there is the ACRF = no mode, where the administrator, at his own risk and risk, writes the ZMIRROR user program for determining the failure of the Primary-server.

ZMIRROR

It is theoretically possible to build such a hardware and software system for the final installation of the mirror, when for any scenario it is possible to unambiguously and programmatically determine the state of the Primary server.
If this situation occurs on your server, it is advisable to set the mode to ACRF = no. Then in the case of a break, and when there is no connection with the Primary-server ISC Agent, the system calls the $$ IsOtherNodeDown ^ ZMIRROR () function. The function returns 1 or 0. In the event that there is no function or the function returns 0, the automatic failover process is terminated.
The task of the programmer when writing a custom implementation of the $$ IsOtherNodeDown ^ ZMIRROR () function is to reliably verify that the Primary server is really out of order and return 1 in this case.



Disclaimer! Using this code in no way guarantees an automatic failover in your production configuration for the ACRF = No mode. For real use of this mode, thorough failover testing is required for each configuration separately!

Mirror Tuning Tips

A few recommendations from InterSystems Database Mirroring operating experience

“You cannot“ manually ”clean the primary-server log if the backup server is turned off”
Indeed, if you do this, and the backup server did not have time to pick up the necessary logs, you can create a backup server again.

“It’s impossible to keep the backup server off for a long time while the primary server is running”
In this case, the disk space of the primary server may run out due to an overflow of unsynchronized logs.

“Time on servers must be synchronized”
This is a simple condition, but it must be met. Otherwise, when out of sync, for example, for a few hours, the backup server gets “confused” with the relevance of the logs. As a result, we make the backup server again.

“The network between the servers must be unloaded”
The network is constantly sending the log from the Primary-server to the backup. If the network between the servers is still busy, or the channel is not enough for data transfer, as a result, the Primary-server will start to “slow down”, which will not have time to transfer data entered by users of the IC.

“System configuration settings, mappings, roles and user settings must be synchronized manually”
Global, class and package mapping settings as well as all settings of roles, users and system configuration for Primary and Backup servers must also be maintained in a synchronized state. Mirror synchronizes only the databases selected for mirroring. Therefore, the administrator of the mirror needs to have identical configuration, mappings and user security policies and maintain this state for Primary and Backup servers.

Performance specifications

The average time for automatic failover for Agent Contact Required for Failover = yes is 10-20 seconds. A pause of 10-20 seconds is the time when the application will continue to work with the new server. In this case, the connection will be reconnected (if there was a direct connection to the server) or simply the continuation of work, if the work was carried out via an ECP connection.
In the case of ACRF = no, the pause averages 35-40 seconds. At the same time, ECP clients will lose the connection, since default timeout for ECP = 30 sec. For systems that use ECP and a mirror with ACRF = no on ECP client servers, in the SYSTEM ^% ZSTART program, it is advisable to write a code that increases the ECP timeout:

do $ system .ECP . SetProperty ( "ECPRecoveryTimout" , 90)

The bottom line or “we need it”?

Answering this question, I say “Yes.” Even if you use what comes out of the box - namely, the ACRF = yes mode, you can perform planned work on updating the OS, Caché, Hardware, and so on. without stopping the operation of the information system, as well as automatically overcoming the failures of the Caché DBMS. The remaining scenarios can be covered with already known means using hardware failover clusters and uninterruptible power systems in the required number of nines.