On the implementation of persistent processes in real-time control systems (part 2)
Continuation of the article.
Start: Part 1
3. Hardware and firmware
')
This level of implementation of the management system is characterized by a very large freedom of choice for the developer. Since we have agreed above not to consider specialized hardware solutions in this article, we restrict ourselves to serial general-purpose equipment.
First of all, in our deep conviction, no serious article on resiliency is unthinkable without paying tribute to IBM and the z Systems and Power Systems platforms. Z Systems mainframes and Power Systems HA clusters are specifically designed to provide a single fault-tolerant platform for user applications at the hardware, firmware and system software level, and in terms of reliability are potentially superior to those solutions that can be implemented on the more common Intel architecture. Unfortunately, these IBM solutions also have certain disadvantages, the most common of which is their cost. Developers' experience shows that, with the modern cost of z, p and Intel solutions (hardware itself and licensed programs for it), and the same with the current dollar / ruble rate, it’s quite difficult in the Russian context to economically justify new investments in proprietary architectures, even with taking into account significant additional labor costs to ensure the specified reliability indicators for Intel solutions. In general, colleagues working with “big iron” know their own reasons well, their way is highly respected, but it cannot be recommended to the beginner.
Considering the Intel-compatible solutions in the field of fault-tolerant hardware, you should pay attention, in particular, to the following points:
- hot backup servers;
- hot backup of network equipment and connections between servers;
- hot backup of disk storage in the data storage system;
- resilience of embedded software;
- control the operating status of the OS.
A convenient ready-made platform for hot backup of servers, network equipment and disk memory are blade and flex systems manufactured by a number of manufacturers. The author is inclined to recommend a similar system to novice developers of fault-tolerant solutions (if the budget allows), since the manufacturer has solved many issues in advance that could otherwise arise only with the acquisition of bitter experience. At the same time, hot sparing can be provided by aggregating separately mounted components. It should be noted that, one way or another, it is necessary to address the issue of organizing a centralized data storage system with alternative access to eliminate a single point of failure when the data access subsystem fails.
Firmware resilience to failures is provided by special UEFI firmware algorithms and service controllers on server platforms. For example, for servers, automatic backup and recovery of hardware firmware, automatic backup of the operating system loader, etc. can be provided.
Fundamentally important for ensuring the resiliency of the computing environment is the use of various kinds of watchdog timers implemented by server platforms. At the application and cluster software level, the IPMI or iTCO watchdog timer provided by the hardware is used. When booting the operating system, a number of server platforms allow you to set your own timers to monitor the success of this process.
Once, the author happened to observe the behavior of the Lenovo blade server, when updating the operating system boot loader on which the failure occurred, and the boot loader configuration files were recorded with an error. The loader started loading the Linux kernel, and then it hung. After some time, the OS boot timeout worked in the server's service processor, and the server rebooted. UEFI server firmware, having found that the previous download attempt failed, independently made a decision to roll back to the previous version of the bootloader from the archive directory, called it, and the system booted safely. So the system, which on the standard platform of the workstation level would be inoperable before manual loading from the repair section, was automatically automatically loaded on the server platform in two steps, until the next update restored the correct bootloader. At the time of initial system setup, the administrator was only required to set the correct load timer value.
The well-known means of ensuring server fault tolerance include ECC memory, combining disk media into redundant RAID arrays, duplicating all server components and server chassis, providing alternative communication paths between components of a server system, etc.
Summarizing, at present a sufficient variety of specialized server systems is presented on the market, whose hardware capabilities for ensuring fault tolerance should not be neglected.
Finally, we note a common misconception - the opinion that the minimum failover cluster should consist of three nodes, since this is the smallest number that provides for redundancy and at the same time ensures majority voting when one of the nodes fails. In fact, the failure or transfer to service of one node out of three leaves the remaining two nodes in a highly unstable state of mutual competition, which is likely to result in their mutual shooting at the slightest communication failure or when putting the third node back into operation, which will begin to install relations with them in turn. Therefore, the actual failover cluster configuration should include at least 4 or 5 nodes (the probability of a 2: 2 vote, and is very low in itself, since it implies simultaneous problems on two nodes, can be eliminated by the asymmetric cluster topology).
4. Host Operating System
The question of choosing a host operating system for fault-tolerant applications reveals an extreme dependence on a number of useful moments. The main factors that play a role in this choice include the following:
- support of physical equipment;
- support of virtualization and clustering environments;
- cost;
- requirements for certification and security.
Fault-tolerant server platforms are declared by the manufacturer to be compatible with only a small number of operating systems. Typically, such systems for Intel-compatible platforms include Windows, Red Hat Enterprise Linux (RHEL), SUSE Linux Enterprise Server (SLES), and VMware ESXi. Installation of other operating systems is possible, but, as a rule, it leads to the lack of full support for critical hardware capabilities (for example, multipath for redundant disk controllers) to ensure fault tolerance.
The choice of Windows as a platform for building a management system essentially means placing control servers in a concentrated, hostile environment of malicious software that is distributed through the hardware of unqualified users. From the point of view of the author, such a choice can hardly be justified .
The main server virtualization tools used today for Unix-compatible systems are VMware ESXi, KVM (RHEL and SLES), Xen (SLES). All these platforms provide clustering of virtual machines (as an option), that is, support for the automatic migration of virtual machines from a failed node to a backup.
According to the functional characteristics of the hypervisor, to date, VMware ESXi occupies a leading position. However, the cost of VMware licenses for a high availability cluster, with its characteristic significant number of processors, can be quite substantial.
KVM and Xen are simpler and cheaper solutions. The advantages of KVM include the greater development of a number of virtual machine interfaces, the advantages of Xen - functioning on a microkernel, which, theoretically, increases the reliability of the hypervisor.
Finally, we note that the certification and security requirements applicable in the industry for which the developed fault-tolerant system is intended can eliminate all of the above, since none of the listed operating systems may fail to meet the requirements of industry standards. In this case, the design will have to dance on the requirements of regulatory documents.
Ending: part 3
Start: Part 1
3. Hardware and firmware
')
This level of implementation of the management system is characterized by a very large freedom of choice for the developer. Since we have agreed above not to consider specialized hardware solutions in this article, we restrict ourselves to serial general-purpose equipment.
First of all, in our deep conviction, no serious article on resiliency is unthinkable without paying tribute to IBM and the z Systems and Power Systems platforms. Z Systems mainframes and Power Systems HA clusters are specifically designed to provide a single fault-tolerant platform for user applications at the hardware, firmware and system software level, and in terms of reliability are potentially superior to those solutions that can be implemented on the more common Intel architecture. Unfortunately, these IBM solutions also have certain disadvantages, the most common of which is their cost. Developers' experience shows that, with the modern cost of z, p and Intel solutions (hardware itself and licensed programs for it), and the same with the current dollar / ruble rate, it’s quite difficult in the Russian context to economically justify new investments in proprietary architectures, even with taking into account significant additional labor costs to ensure the specified reliability indicators for Intel solutions. In general, colleagues working with “big iron” know their own reasons well, their way is highly respected, but it cannot be recommended to the beginner.
Note for the public sector
Here we are forced to make a fork in the logic of our presentation, and take into account the fact that a significant share of the market for fault-tolerant systems in Russia is focused on the needs of the public sector. Therefore, for developers burdened, apart from other concerns, with honorary duties to serve the wishes of the state, we note the following. At the present time, as is known, the Government of the Russian Federation has announced an import substitution policy. In its most fundamental form, this policy implies the exclusive use of products manufactured in Russia and the EEU countries. However, a number of governing documents establish softer requirements that dictate a restriction on the use of products only by NATO countries, the European Union and others that support the regime of sectoral sanctions against the Russian Federation. For the information technology industry, it is essential that the PRC (including Taiwan) and Japan do not fall under the mode of such soft restrictions, which brings the server systems of Lenovo companies (quite successfully outbid IBM-compatible business IBM) and Hitachi to the front lines for consideration.
Considering the Intel-compatible solutions in the field of fault-tolerant hardware, you should pay attention, in particular, to the following points:
- hot backup servers;
- hot backup of network equipment and connections between servers;
- hot backup of disk storage in the data storage system;
- resilience of embedded software;
- control the operating status of the OS.
A convenient ready-made platform for hot backup of servers, network equipment and disk memory are blade and flex systems manufactured by a number of manufacturers. The author is inclined to recommend a similar system to novice developers of fault-tolerant solutions (if the budget allows), since the manufacturer has solved many issues in advance that could otherwise arise only with the acquisition of bitter experience. At the same time, hot sparing can be provided by aggregating separately mounted components. It should be noted that, one way or another, it is necessary to address the issue of organizing a centralized data storage system with alternative access to eliminate a single point of failure when the data access subsystem fails.
Firmware resilience to failures is provided by special UEFI firmware algorithms and service controllers on server platforms. For example, for servers, automatic backup and recovery of hardware firmware, automatic backup of the operating system loader, etc. can be provided.
Fundamentally important for ensuring the resiliency of the computing environment is the use of various kinds of watchdog timers implemented by server platforms. At the application and cluster software level, the IPMI or iTCO watchdog timer provided by the hardware is used. When booting the operating system, a number of server platforms allow you to set your own timers to monitor the success of this process.
Once, the author happened to observe the behavior of the Lenovo blade server, when updating the operating system boot loader on which the failure occurred, and the boot loader configuration files were recorded with an error. The loader started loading the Linux kernel, and then it hung. After some time, the OS boot timeout worked in the server's service processor, and the server rebooted. UEFI server firmware, having found that the previous download attempt failed, independently made a decision to roll back to the previous version of the bootloader from the archive directory, called it, and the system booted safely. So the system, which on the standard platform of the workstation level would be inoperable before manual loading from the repair section, was automatically automatically loaded on the server platform in two steps, until the next update restored the correct bootloader. At the time of initial system setup, the administrator was only required to set the correct load timer value.
The well-known means of ensuring server fault tolerance include ECC memory, combining disk media into redundant RAID arrays, duplicating all server components and server chassis, providing alternative communication paths between components of a server system, etc.
Summarizing, at present a sufficient variety of specialized server systems is presented on the market, whose hardware capabilities for ensuring fault tolerance should not be neglected.
Finally, we note a common misconception - the opinion that the minimum failover cluster should consist of three nodes, since this is the smallest number that provides for redundancy and at the same time ensures majority voting when one of the nodes fails. In fact, the failure or transfer to service of one node out of three leaves the remaining two nodes in a highly unstable state of mutual competition, which is likely to result in their mutual shooting at the slightest communication failure or when putting the third node back into operation, which will begin to install relations with them in turn. Therefore, the actual failover cluster configuration should include at least 4 or 5 nodes (the probability of a 2: 2 vote, and is very low in itself, since it implies simultaneous problems on two nodes, can be eliminated by the asymmetric cluster topology).
4. Host Operating System
The question of choosing a host operating system for fault-tolerant applications reveals an extreme dependence on a number of useful moments. The main factors that play a role in this choice include the following:
- support of physical equipment;
- support of virtualization and clustering environments;
- cost;
- requirements for certification and security.
Fault-tolerant server platforms are declared by the manufacturer to be compatible with only a small number of operating systems. Typically, such systems for Intel-compatible platforms include Windows, Red Hat Enterprise Linux (RHEL), SUSE Linux Enterprise Server (SLES), and VMware ESXi. Installation of other operating systems is possible, but, as a rule, it leads to the lack of full support for critical hardware capabilities (for example, multipath for redundant disk controllers) to ensure fault tolerance.
The choice of Windows as a platform for building a management system essentially means placing control servers in a concentrated, hostile environment of malicious software that is distributed through the hardware of unqualified users. From the point of view of the author, such a choice can hardly be justified .
The main server virtualization tools used today for Unix-compatible systems are VMware ESXi, KVM (RHEL and SLES), Xen (SLES). All these platforms provide clustering of virtual machines (as an option), that is, support for the automatic migration of virtual machines from a failed node to a backup.
According to the functional characteristics of the hypervisor, to date, VMware ESXi occupies a leading position. However, the cost of VMware licenses for a high availability cluster, with its characteristic significant number of processors, can be quite substantial.
KVM and Xen are simpler and cheaper solutions. The advantages of KVM include the greater development of a number of virtual machine interfaces, the advantages of Xen - functioning on a microkernel, which, theoretically, increases the reliability of the hypervisor.
Finally, we note that the certification and security requirements applicable in the industry for which the developed fault-tolerant system is intended can eliminate all of the above, since none of the listed operating systems may fail to meet the requirements of industry standards. In this case, the design will have to dance on the requirements of regulatory documents.
Ending: part 3
Source: https://habr.com/ru/post/303296/
All Articles