Creating a reliable storage distributed to multiple servers on nfs
When creating a cluster for processing calls based on CGP , it became necessary to configure uninterrupted storage mounted from several servers.
Ubuntu Server 10.10 was taken as a distribution for servers. The disk space was divided into two logical disks (sda1 for the installation of the system, and sda2 for the shared disk itself).
After installing the base system, you must additionally install the following packages: heartbeat, pacemaker, drbd8-utils, xfs, xfsprogs, nfs-kernel-server.
')
Heartbeat and pacemaker are needed for server clustering. The storage is based on drbd, xfs was used as the file system. The distribution of the file system servers made by nfs.
The names u1 and u2 were chosen for the nodes. For convenience, these names were immediately registered in / etc / hosts:
u0 is the address at which the storage is available for mounting the file system from other servers.
The file with the storage configuration is located in /etc/drbd.d/r0.res:
On both nodes, the file is the same, so you can create it on one and then copy to the second.
Such a brutal setting for the treatment of the split brain was chosen, since the storage is mainly used to store the system configuration. That is, the loss of recent changes is not as critical as the loss of calls during downtime due to slit blain.
After creating the configuration file, you must create the disks themselves on both servers in the cluster:
After that you can start drbd. It is important to start the drbd daemon on both servers with a difference of less than a minute (timeout for communicating with other cluster members):
After this, in / proc / drbd will be the state of the storage, in approximately the form:
That is, two nodes in the secondary mode, with the disc inconsistent. To get out of this position, you must forcibly declare one of the nodes of the cluster as the main one; to do this, you must execute the command:
After that drbd will start updating the state of the secondary disk:
After the synchronization is complete, you can create a file system on the server, where drbd is in the primary state:
For the next steps we will be hampered by the standard daemon launch mechanism. Therefore, you need to run the command on both servers:
The configuration is created in the /etc/heartbeat/ha.cf file. It is the same on both nodes, so you can create it on one and then copy it to the second.
The second file is used to authenticate / etc / heartbeat / authkeys:
It is better not to wait for a warning from heartbeat to the wrong file attributes and change them in advance:
After that, you can start heartbeat:
After some time, the crm_mod command should show that two nodes have connected to each other:
Next, you need to run the crm configure edit command and enter the settings for the cluster:
The exportfs settings indicate that the directory where the shared file system will be mounted, this resource can be mounted over nfs to all servers from the 10.0.0.0 network.
After some time, the cluster monitor should show the status of resources:
You can use / etc / fstab to do this:
Ubuntu Server 10.10 was taken as a distribution for servers. The disk space was divided into two logical disks (sda1 for the installation of the system, and sda2 for the shared disk itself).
After installing the base system, you must additionally install the following packages: heartbeat, pacemaker, drbd8-utils, xfs, xfsprogs, nfs-kernel-server.
')
Heartbeat and pacemaker are needed for server clustering. The storage is based on drbd, xfs was used as the file system. The distribution of the file system servers made by nfs.
1. System Setup
The names u1 and u2 were chosen for the nodes. For convenience, these names were immediately registered in / etc / hosts:
10.0.0.84 u1 10.0.0.115 u2 10.0.0.120 u0
u0 is the address at which the storage is available for mounting the file system from other servers.
2. Setup drbd
The file with the storage configuration is located in /etc/drbd.d/r0.res:
resource r0 {
protocol C;
syncer {
rate 4M;
}
startup {
wfc-timeout 15;
degr-wfc-timeout 60;
}
net {
after-sb-0pri discard-zero-changes;
after-sb-1pri discard-secondary;
after-sb-2pri disconnect;
cram-hmac-alg sha1;
shared-secret somesecretword;
}
on u1 {
device / dev / drbd0;
disk / dev / sda2;
address 10.0.0.84:7788;
meta-disk internal;
}
on u2 {
device / dev / drbd0;
disk / dev / sda2;
address 10.0.0.115:7788;
meta-disk internal;
}
}
On both nodes, the file is the same, so you can create it on one and then copy to the second.
Such a brutal setting for the treatment of the split brain was chosen, since the storage is mainly used to store the system configuration. That is, the loss of recent changes is not as critical as the loss of calls during downtime due to slit blain.
After creating the configuration file, you must create the disks themselves on both servers in the cluster:
dd if = / dev / zero of = / dev / sda2 bs = 64M drbdadm create-md r0
After that you can start drbd. It is important to start the drbd daemon on both servers with a difference of less than a minute (timeout for communicating with other cluster members):
/etc/init.d/drbd start
After this, in / proc / drbd will be the state of the storage, in approximately the form:
0: cs: Connected ro: Secondary / Secondary ds: Inconsistent / Inconsistent C r ----
That is, two nodes in the secondary mode, with the disc inconsistent. To get out of this position, you must forcibly declare one of the nodes of the cluster as the main one; to do this, you must execute the command:
drbdadm - --overwrite-data-of-peer primary r0
After that drbd will start updating the state of the secondary disk:
0: cs: SyncSource ro: Primary / Secondary ds: UpToDate / Inconsistent C r ----
ns: 241984 nr: 0 dw: 0 dr: 242184 al: 0 bm: 14 lo: 510 pe: 179 ua: 510 ap: 0 ep: 1 wo: b oos: 782664
[===> ................] sync'ed: 23.6% (782664/1023932) K
finish: 0:04:04 speed: 3,160 (3,172) K / sec
After the synchronization is complete, you can create a file system on the server, where drbd is in the primary state:
mkfs.xfs / dev / drbd0
For the next steps we will be hampered by the standard daemon launch mechanism. Therefore, you need to run the command on both servers:
update-rc.d -f drbd remove
3. Configure heartbeat
The configuration is created in the /etc/heartbeat/ha.cf file. It is the same on both nodes, so you can create it on one and then copy it to the second.
logfacility daemon keepalive 2 deadtime 15 warntime 5 initdead 120 udpport 694 ucast eth0 10.0.0.115 ucast eth0 10.0.0.84 auto_failback on node u1 node u2 use_logd yes crm respawn
The second file is used to authenticate / etc / heartbeat / authkeys:
auth 1 1 sha1 somesecretword
It is better not to wait for a warning from heartbeat to the wrong file attributes and change them in advance:
chmod 600 / etc / heartbeat / authkeys
After that, you can start heartbeat:
/etc/init.d/heartbeat start
After some time, the crm_mod command should show that two nodes have connected to each other:
============ Last updated: Fri Feb 10 09:33:04 2012 Stack: Heartbeat Current DC: u1 (86b204d8-ee3e-47c7-ba0e-1dcbd40a20da) - partition with quorum Version: 1.0.9-unknown 2 Nodes configured, 2 expected votes 2 Resources configured. ============ Online: [u2 u1]
Next, you need to run the crm configure edit command and enter the settings for the cluster:
node $ id = "86b204d8-ee3e-47c7-ba0e-1dcbd40a20da" u1
node $ id = "c6e3c21f-da3e-4031-9f28-a7e33425a817" u2
primitive drbd0 ocf: linbit: drbd \
params drbd_resource = "r0" \
op start interval = "0" timeout = "240" \
op stop interval = "0" timeout = "100" \
op monitor interval = "20" role = "Slave" timeout = "20" depth = "0" \
op monitor interval = "10" role = "Master" timeout = "20" depth = "0"
primitive fs0 ocf: heartbeat: Filesystem \
params directory = "/ shared" fstype = "xfs" device = "/ dev / drbd / by-res / r0" options = "noatime, nodiratime, nobarrier, logbufs = 8" \
op start interval = "0" timeout = "60" \
op stop interval = "0" timeout = "60" \
op notify interval = "0" timeout = "60" \
op monitor interval = "20" timeout = "40" depth = "0" \
meta target-role = "Started"
primitive ip0 ocf: heartbeat: ipaddr2 \
params ip = "10.0.0.120" nic = "eth0: 0" \
op monitor interval = "5s" \
meta target-role = "Started"
primitive nfs0 ocf: itl: exportfs \
params directory = "/ shared" clientspec = "10.0.0.0/255.255.255.0" options = "rw, no_root_squash, sync, no_wdelay" fsid = "1" \
op start interval = "0" timeout = "40" \
op stop interval = "0" timeout = "60" \
op monitor interval = "30" timeout = "55" depth = "0" OCF_CHECK_LEVEL = "10" \
meta target-role = "Started"
group ha_nfs fs0 nfs0 ip0 \
meta target-role = "Started"
ms ms_drbd0 drbd0 \
meta master-max = "1" master-node-max = "1" clone-max = "2" clone-node-max = "1" notify = "true"
colocation c_nfs inf: nfs0 ms_drbd0: Master
order o_nfs inf: ms_drbd0: promote ha_nfs: start
property $ id = "cib-bootstrap-options" \
dc-version = "1.0.9-unknown" \
cluster-infrastructure = "Heartbeat" \
stonith-enabled = "false" \
expected-quorum-votes = "2" \
no-quorum-policy = "ignore" \
symmetric-cluster = "true" \
last-lrm-refresh = "1328625786"
rsc_defaults $ id = "rsc_defaults-options" \
resource-stickiness = "10,000"
The exportfs settings indicate that the directory where the shared file system will be mounted, this resource can be mounted over nfs to all servers from the 10.0.0.0 network.
After some time, the cluster monitor should show the status of resources:
============
Last updated: Fri Feb 10 09:33:04 2012
Stack: Heartbeat
Current DC: u1 (86b204d8-ee3e-47c7-ba0e-1dcbd40a20da) - partition with quorum
Version: 1.0.9-unknown
2 Nodes configured, 2 expected votes
2 Resources configured.
============
Online: [u2 u1]
Resource Group: ha_nfs
fs0 (ocf :: heartbeat: Filesystem): Started u1
nfs0 (ocf :: itl: exportfs): Started u1
ip0 (ocf :: heartbeat: ipaddr2): Started u1
Master / Slave Set: ms_drbd0
Masters: [u1]
Slaves: [u2]
4. Mounting a shared file system from other servers
You can use / etc / fstab to do this:
u0: / shared / var / CommuniGate / SharedDomains nfs bg, intr 0 0
5. Additional links
- When creating the configuration, the description was taken as the basis http://library.linode.com/linux-ha/ip-failover-heartbeat-pacemaker-drbd-mysql-ubuntu-10.04
- Treatment instructions for split-brain from drbd http://www.alsigned.ru/?p=490
- In Ubuntu 10.10 exportfs, the agent is not included, so you need to download it separately ( https://github.com/ClusterLabs/resource-agents/blob/master/heartbeat/exportfs ) and install it in /usr/lib/ocf/resource.d/ heartbeat /
- There is no important part in this configuration - STONITH ( http://linux-ha.org/wiki/STONITH ), since there is no backup channel
- A description of the mechanism for using a common address can be found at http://www.ultramonkey.org/3/ip_address_takeover.html
Source: https://habr.com/ru/post/137938/
All Articles