HA cluster, file systems replicated over the network
What: did a high availability cluster on two nodes using heartbeat. Cluster for web server (apache, nginx, php, mysql). This is not an instruction on raising such a cluster, but notes on the use of cluster file systems, something that is missing in common articles and a description of the rake that I stepped on.
First, what is missing in the description of drbd setup (http://www.opennet.ru/base/sys/drbd_setup.txt.html):
Setting drbd to shrink the file system on an existing partition - use the command
The desired partition will be issued by drbd itself, indicated in kilobytes (for example, 1000K)
After starting the disk, it will give to / proc / drbd something like this
When entering a command
will sophistically swear type
in old articles it is recommended to do
but he does not understand such simple and accessible keys and there is nothing in this mana about this, but the right way to do it is:
Then happiness comes and he begins to sync the disk, which is seen in / proc / drbd like this:
Now about glusterfs:
A wonderful file system ... at first it seemed, replication was a master-master, a lot of lotions that can be combined, and the main difference from drbd is that it can be mounted on all the nodes simultaneously.
The jamb number 1, according to the developers, should be fixed in version 2.0.1 (did not check if it was fixed correctly) - using the glusterfs section to store the mysql database is contraindicated! Mysql puts locks on the database files and does not remove them immediately after the completion of the work or, for example, the death of the node. And when mysql from the second node tries to work with this database, the entire node starts to blunt on black due to the glusterfsd server process and as a result the cluster is not functional.
Jamb number 2 - performance. I will not say for other configurations, but for a replicable partition into two nodes, with the configuration from the example on the site of the cluster, the performance of Apache (the whole www lies on the cluster section) drops to 10 times. The measurement was performed by the ab utility with the number of competing requests 10. Through long experimentation with configs, the best config of the client was identified (this is for my case when two nodes share their sections). In the example, first both sections were connected via the network, then they were united into mirror, after which the cache and thread translators were applied to this mirror. With this option, the performance is 10 times worse than with the direct operation of the Apache with the disks. If we change the config as follows: we catch the current node's partition through posix (as well as the server does it), the remote partition as in the example through the network, then we apply the cache to the remote partition and then we collect the volume and local partition into myrrh. The threads only slow down the work, reading ahead does not give results, the delayed write in my example was not needed, since the record very rarely takes place. In this configuration, the performance loss relative to the use of the local partition is only about 50%. But because of this, I refused glusterfs in favor of the second section on drbd (the first one was tuned to musculle, and the second is mounted on the second node under apache). I also want to note that in tests of direct reading, glusterfs practically does not show the difference with local file systems, but in my case ... alas.
First, what is missing in the description of drbd setup (http://www.opennet.ru/base/sys/drbd_setup.txt.html):
Setting drbd to shrink the file system on an existing partition - use the command
resize2fs <__> <_>The desired partition will be issued by drbd itself, indicated in kilobytes (for example, 1000K)
After starting the disk, it will give to / proc / drbd something like this
cs:Connected st:Secondary/Secondary ds:Inconsistent/InconsistentWhen entering a command
drbdadm primary <_>will sophistically swear type
/dev/drbd1: State change failed: (-2) Refusing to be Primary without at least one UpToDate disk
Command 'drbdsetup /dev/drbd1 primary' terminated with exit code 17in old articles it is recommended to do
drbdadm -- --do-what-I-say primary <_>but he does not understand such simple and accessible keys and there is nothing in this mana about this, but the right way to do it is:
drbdadm -- --overwrite-data-of-peer primary <_>Then happiness comes and he begins to sync the disk, which is seen in / proc / drbd like this:
1: cs:SyncSource st:Primary/Secondary ds:UpToDate/Inconsistent C r---
ns:225808 nr:0 dw:0 dr:225808 al:0 bm:13 lo:0 pe:895 ua:0 ap:0
[>....................] sync'ed: 0.4% (71460/71676)M
finish: 0:27:25 speed: 44,444 (44,444) K/sec
resync: used:0/61 hits:111995 misses:14 starving:0 dirty:0 changed:14
act_log: used:0/127 hits:0 misses:0 starving:0 dirty:0 changed:0Now about glusterfs:
A wonderful file system ... at first it seemed, replication was a master-master, a lot of lotions that can be combined, and the main difference from drbd is that it can be mounted on all the nodes simultaneously.
The jamb number 1, according to the developers, should be fixed in version 2.0.1 (did not check if it was fixed correctly) - using the glusterfs section to store the mysql database is contraindicated! Mysql puts locks on the database files and does not remove them immediately after the completion of the work or, for example, the death of the node. And when mysql from the second node tries to work with this database, the entire node starts to blunt on black due to the glusterfsd server process and as a result the cluster is not functional.
Jamb number 2 - performance. I will not say for other configurations, but for a replicable partition into two nodes, with the configuration from the example on the site of the cluster, the performance of Apache (the whole www lies on the cluster section) drops to 10 times. The measurement was performed by the ab utility with the number of competing requests 10. Through long experimentation with configs, the best config of the client was identified (this is for my case when two nodes share their sections). In the example, first both sections were connected via the network, then they were united into mirror, after which the cache and thread translators were applied to this mirror. With this option, the performance is 10 times worse than with the direct operation of the Apache with the disks. If we change the config as follows: we catch the current node's partition through posix (as well as the server does it), the remote partition as in the example through the network, then we apply the cache to the remote partition and then we collect the volume and local partition into myrrh. The threads only slow down the work, reading ahead does not give results, the delayed write in my example was not needed, since the record very rarely takes place. In this configuration, the performance loss relative to the use of the local partition is only about 50%. But because of this, I refused glusterfs in favor of the second section on drbd (the first one was tuned to musculle, and the second is mounted on the second node under apache). I also want to note that in tests of direct reading, glusterfs practically does not show the difference with local file systems, but in my case ... alas.
')
Source: https://habr.com/ru/post/60974/
All Articles