Linux HA based on Pacemaker
In my previous article, I briefly touched upon the topic of creating a High Availability solution based on the heartbeat daemon. However, as it turned out, something more complicated than a 2-node cluster on it is not so convenient to do. Studying the problem put me on the trail of the Pacemaker project. We will now look at it briefly.
According to the official documentation, Pacemaker is a cluster resource manager with the following main features:
The project, in addition to its own demons that support the cluster configuration and the above shell, also uses third-party, the same heartbeat and corosync. I personally (the authors themselves, too) recommend using corosync. In general, it supports heartbeat scripts, so there should be no problems.
There is nothing easier on RHEL / CentOS.
Then we edit /etc/corosync/corosync.conf (there is corosync.conf.sample) and start.
Yes, it is imperative to have a properly configured DNS so that all nodes relocate in both directions.
How to add nodes? There is nothing easier, put Pacemaker, copy /etc/corosync/corosync.conf and / etc / ais / authkeys (if configured) and run. The node will be automatically included in the cluster. The cluster configuration will also be automatically copied.
')
If the node has failed and you replaced the iron? The same, giving the new node the same name, copying the configs and starting. Lepot
Cluster configuration is a simple XML. However, manually edit it NOT. All changes to the config are subject to automatic versioning, the nodes do not receive the entire config at once, but only the deltas from what they know about. Those. if the node has failed, and at this time you have changed the config, then when it returns, it will receive all the changes you have made.
To work with the config, you can use either cibadmin or the crm shell itself. You can see the config for example:
In addition to the cluster options themselves, configuration also includes:
How to add nodes, we have already figured out. Nodes are removed much more difficult: stop corosync on the node, and then remove the node from the config.
Do not forget that quorum is reached when more than half of the nodes are in the ranks. Therefore, if you have a cluster of only 2, then this option should be disabled, otherwise if any of them fall, the cluster will consider itself collapsed.
What is a resource in terms of corosync? Anything that can be scripted! Usually, scripts are written in bash, but nothing prevents you from writing them in Perl, Python, or even C. All that the script needs is to perform 3 actions: start, stop and monitor. In general, scripts must comply with LSB (Linux Standard Base) or OCF (Open Cluster Framework) - the latter expands the LSB somewhat, also requiring the transfer of parameters through environment variables with a special name.
As we can see, there are quite a few ready-made agents (Resource Agents). However, it is unlikely that it will be very difficult to write the new one yourself.
When creating a resource, we will need to specify its class, type, provider, and the actual name with additional parameters. In the list above: ocf - class, heartbeat - provider, IPaddr - agent type.
Resources support many additional parameters, like binding to a node (resource-stickiness), default roles (started, stoped, master), etc. There are opportunities to create groups of resources, clones (working on several nodes), etc.
To begin with, any link has its own weight - an integer ranging from -INFINITY to + INFINITY. Moreover, if the weight of the bond is ± INFINITY, then it is considered rigid, otherwise - soft, i.e. if the weights of other links are higher, it can be ignored.
Relationships determine the binding of resources to a site (location), the order in which resources are started (ordering) and their co-residence on a site (colocation).
Here we created another resource - WebSite, set it to live with ClusterIP (by the way, -INFINITY would mean that resources should NOT be on the same node), determine the launch order: first ClusterIP, then WebSite, and then set that WebSite is very it is desirable to be on the pcmk-1 node, with a weight of 50. When setting the order, you can also specify whether you can start resources in parallel or do it sequentially.
In general, this way you can build quite complex chains of resource dependencies. Everything becomes even more interesting if we consider that it is possible to set more complex rules (rules) that work, for example, only at a certain time of day or depending on the state of the cluster components.
By the way, to facilitate understanding, the authors gave Pacemaker the opportunity to build graphviz graphics - see the ptest utility.
Pacemaker is simple and convenient, but at the same time very powerful and rich in capabilities. After IBM HA MS, I can't get enough of this project at all :). In order not to score a post with unnecessary details, and for further study you will need the following literature:
Thanks for attention.
Posemaker
According to the official documentation, Pacemaker is a cluster resource manager with the following main features:
- Detection and recovery of failures at the level of nodes and services;
- Independence from the storage subsystem: a shared disk is not required;
- Resource type independence: everything that can be scripted can be clustered;
- Support STONITH (Shoot-The-Other-Node-In-The-Head) - drugs from Split-Brain;);
- Support for clusters of any size;
- Support for both quorum and resource-dependent clusters;
- Support virtually any redundant configuration;
- Automatic replication of the config to all nodes of the cluster;
- Ability to set the order of start-up resources, as well as their compatibility on one node
- Support for advanced resource types: clones (running on multiple nodes) and with additional states (master / slave, etc.);
- Single cluster shell (crm), unified, scripted.
The project, in addition to its own demons that support the cluster configuration and the above shell, also uses third-party, the same heartbeat and corosync. I personally (the authors themselves, too) recommend using corosync. In general, it supports heartbeat scripts, so there should be no problems.
Installation
There is nothing easier on RHEL / CentOS.
cd /etc/yum.repos.d wget http://www.clusterlabs.org/rpm/epel-5/clusterlabs.repo yum install pacemaker Then we edit /etc/corosync/corosync.conf (there is corosync.conf.sample) and start.
/etc/init.d/corosync start Yes, it is imperative to have a properly configured DNS so that all nodes relocate in both directions.
How to add nodes? There is nothing easier, put Pacemaker, copy /etc/corosync/corosync.conf and / etc / ais / authkeys (if configured) and run. The node will be automatically included in the cluster. The cluster configuration will also be automatically copied.
')
If the node has failed and you replaced the iron? The same, giving the new node the same name, copying the configs and starting. Lepot
# crm_mon ============ Last updated: Thu Aug 27 16:54:55 2009 Stack: openais Current DC: pcmk-1 - partition with quorum Version: 1.0.5-462f1569a43740667daf7b0f6b521742e9eb8fa7 2 Nodes configured, 2 expected votes 0 Resources configured. ============ Online: [ pcmk-1 pcmk-2 ] Configuration
Cluster configuration is a simple XML. However, manually edit it NOT. All changes to the config are subject to automatic versioning, the nodes do not receive the entire config at once, but only the deltas from what they know about. Those. if the node has failed, and at this time you have changed the config, then when it returns, it will receive all the changes you have made.
To work with the config, you can use either cibadmin or the crm shell itself. You can see the config for example:
# crm configure show node pcmk-1 node pcmk-2 property $id="cib-bootstrap-options" \ dc-version="1.0.5-462f1569a43740667daf7b0f6b521742e9eb8fa7" \ cluster-infrastructure="openais" \ expected-quorum-votes="2" In addition to the cluster options themselves, configuration also includes:
- Nodes;
- Resources (Resources);
- Connections (Constraints).
Knots
How to add nodes, we have already figured out. Nodes are removed much more difficult: stop corosync on the node, and then remove the node from the config.
Do not forget that quorum is reached when more than half of the nodes are in the ranks. Therefore, if you have a cluster of only 2, then this option should be disabled, otherwise if any of them fall, the cluster will consider itself collapsed.
Resources
What is a resource in terms of corosync? Anything that can be scripted! Usually, scripts are written in bash, but nothing prevents you from writing them in Perl, Python, or even C. All that the script needs is to perform 3 actions: start, stop and monitor. In general, scripts must comply with LSB (Linux Standard Base) or OCF (Open Cluster Framework) - the latter expands the LSB somewhat, also requiring the transfer of parameters through environment variables with a special name.
# crm ra classes heartbeat lsb ocf / heartbeat pacemaker stonith # crm ra list ocf heartbeat AoEtarget AudibleAlarm ClusterMon Delay Dummy EvmsSCC Evmsd Filesystem ICP IPaddr IPaddr2 IPsrcaddr LVM LinuxSCSI MailTo ManageRAID ManageVE Pure-FTPd Raid1 Route SAPDatabase SAPInstance SendArp ServeRAID SphinxSearchDaemon Squid Stateful SysInfo VIPArip VirtualDomain WAS WAS6 WinPopup Xen Xinetd anything apache db2 drbd eDir88 iSCSILogicalUnit iSCSITarget ids iscsi ldirectord mysql mysql-proxy nfsserver oracle oralsnr pgsql pingd portblock rsyncd scsi2reservation sfex tomcat vmware As we can see, there are quite a few ready-made agents (Resource Agents). However, it is unlikely that it will be very difficult to write the new one yourself.
When creating a resource, we will need to specify its class, type, provider, and the actual name with additional parameters. In the list above: ocf - class, heartbeat - provider, IPaddr - agent type.
crm configure primitive ClusterIP ocf:heartbeat:IPaddr2 \ params ip=192.168.9.101 cidr_netmask=32 \ op monitor interval=30s Resources support many additional parameters, like binding to a node (resource-stickiness), default roles (started, stoped, master), etc. There are opportunities to create groups of resources, clones (working on several nodes), etc.
Connections
To begin with, any link has its own weight - an integer ranging from -INFINITY to + INFINITY. Moreover, if the weight of the bond is ± INFINITY, then it is considered rigid, otherwise - soft, i.e. if the weights of other links are higher, it can be ignored.
Relationships determine the binding of resources to a site (location), the order in which resources are started (ordering) and their co-residence on a site (colocation).
# crm configure primitive WebSite ocf:heartbeat:apache \ params configfile=/etc/httpd/conf/httpd.conf \ op monitor interval=1min # crm configure colocation website-with-ip INFINITY: WebSite ClusterIP # crm configure order apache-after-ip mandatory: ClusterIP WebSite # crm configure location prefer-pcmk-1 WebSite rule 50: pcmk-1 Here we created another resource - WebSite, set it to live with ClusterIP (by the way, -INFINITY would mean that resources should NOT be on the same node), determine the launch order: first ClusterIP, then WebSite, and then set that WebSite is very it is desirable to be on the pcmk-1 node, with a weight of 50. When setting the order, you can also specify whether you can start resources in parallel or do it sequentially.
In general, this way you can build quite complex chains of resource dependencies. Everything becomes even more interesting if we consider that it is possible to set more complex rules (rules) that work, for example, only at a certain time of day or depending on the state of the cluster components.
By the way, to facilitate understanding, the authors gave Pacemaker the opportunity to build graphviz graphics - see the ptest utility.
Conclusion
Pacemaker is simple and convenient, but at the same time very powerful and rich in capabilities. After IBM HA MS, I can't get enough of this project at all :). In order not to score a post with unnecessary details, and for further study you will need the following literature:
- Cluster from Scratch - a detailed example of installation and configuration with brief explanations;
- Pacemaker Explained - a complete, exhaustive description of all settings with examples;
- CRM command line interface - crm shell description;
- clusterlabs.org is the project site itself.
Thanks for attention.
Source: https://habr.com/ru/post/107837/
All Articles