csync2 or how to facilitate the work with the cluster

Not so long ago, I had to raise a Linux cluster for one fairly loaded project. Rather, the issue of resiliency was more important than the load, but usually the cluster is designed to solve both of these problems at the same time.

In this case, I'm not going to consider the cluster architecture or debugging nuances, but tell you about a very convenient way to manage the cluster, speeding up its configuration and debugging.



Agree, it is convenient to have a set of files (for example, configs), which will always look the same on servers with the same role? Under the cut, I'll tell you how to achieve this in the shortest possible time.



In the described cluster, seven dedicated servers that communicate with each other over the internal network with the following hostname:



Load Balancers lb1 lb2

Application Servers app1 app2

Database Servers db1 db2

Backup server bckp1



And the hero of today's story is csync2 , a rather old program that is available in many nix repositories, and can also be downloaded as a tarball or from a git repository .

')

So, step by step guide to installing, configuring and deriving benefit from csync2.

Installation

The installation must be performed on all nodes of the cluster:

From debian \ ubuntu repositories

apt-get install csync2 -y

From source

Before installing, make sure that you have librsync on your system

Otherwise, you need to download librsync-0.9.7.tar.gz ,

and install it

tar -xf librsync-0.9.7.tar.gz && cd librsync-0.9.7

./configure && make

make install



Also, csync2 hopes for libsqlite , which can also be compiled from source codes:

wget www.sqlite.org/sqlite-autoconf-3070603.tar.gz && tar -xf sqlite-autoconf-3070603.tar.gz && cd sqlite-autoconf-3070603

./configure && make

make install



Or you can use a rather unusual way, specifying the path to the tarball with the library when configuring (./configure) csync2:

./configure --with-libsqlite-source=/path/to/libsqlite.tar.gz



I did not receive any other requirements, and due to age, the program has precompiled packages for most OSs ( RedHat RPM , apt-get \ aptitude install csync2, FreeBSD ports )



Compile csync2 itself

cd /usr/local/src && wget oss.linbit.com/csync2/csync2-1.34.tar.gz

tar -xf csync2-1.34.tar.gz && cd csync2-1.34

./configure && make

make install





I performed all operations under Ubuntu 10.04 LTS, where csync2 is installed in one line.

If something went wrong in your OS - write in the comments, I will try to help.

Primary setup

So, we have installed csync2 on all nodes, we need to connect them together and make them exchange files with each other.



csync2 exchanges files through an encrypted SSL connection, so you need to create a single csync2 certificate that will allow servers to “trust” each other:



According to the instructions, you can execute the command in the source folder.

make cert

or (as I did) generate the certificate manually:

openssl genrsa -out /etc/csync2_ssl_key.pem 1024

openssl req -new -key /etc/csync2_ssl_key.pem -out /etc/csync2_ssl_cert.csr

openssl x509 -req -days 600 -in /etc/csync2_ssl_cert.csr -signkey /etc/csync2_ssl_key.pem -out /etc/csync2_ssl_cert.pem



After that, you need to start generating the csync2 key:

csync2 -k /etc/csync2.cluster.key



This command takes quite a long time, after which the file /etc/csync2.cluster.key appears



Now you need to copy it to all the nodes of your cluster so that they will be in a single sync cloud. You can create several keys so that, for example, the database servers cannot communicate with the Application servers, but in my opinion it is not at all necessary if you do not build a cluster for the bank.

First sync

The most important program file is /etc/csync2.cnf

It works on the following principle.

You specify logical groups of servers and indicate what they have in common?

For example, I use internal addressing not by IP, but by hostname, respectively, I want the / etc / hosts file on all machines to be the same, and when adding a new node, it was enough for me to change it once, all changes would leak to the rest of the cluster nodes and they knew who app3 was , for example.

On LoadBalancers, I have nginx, which should also have the same config on different machines.



So in the config file, I combine my servers into logical groups:

# All servers synchronize the base set of configs

group all {

# IP,

host app1 app2;

host db1 db2;

host lb1 lb2;

host bckp1;



#

key /etc/csync2.cluster.key;



# \ ?

include /etc/hosts; # hosts

include /etc/csync2.cfg; # ! csync2 ! ;)



auto younger;# ? -

}



# LoadBalancer' nginx

group lb {

host lb1 lb2;



key /etc/csync2.cluster.key;



include /etc/nginx/*;

auto younger;

}



Now run csync2 with the instruction to synchronize everything that can be synchronized:

csync2 -x



After the initial authorization, all hosts are synchronized and the files you specify will be the same on all nodes.

Possible problems

If something went wrong, run

csync2 -xv

csync2 -xvv

etc.



I csync2 swore at the lack of access to the / etc / hosts entry on other machines, this problem resolved itself after csync2 was also running on them for the first time.

If csync2 swears on SSL - check whether the file /etc/csync2.cluster.key is copied to all hosts and correctly specified in the config.



I didn’t have any other problems, if you have something else - write, let's see.

Goodies

Synchronization of all necessary configs is cool, I put configs from mysql, php, nginx and so on in csync2. It turned out to be very convenient to put the config from csync2 into csync2 itself (almost recursion).

But just syncing files is not everything.

After changing the nginx files, you must restart it! And if it needs to be done on a hundred machines?



We add to the group of servers with the same role on which nginx is installed:

group lb {

host lb1;

host lb2;

key /etc/csync2.cluster.key;



include /etc/nginx; #

action {

pattern /etc/nginx/*; # , ?

exec "/etc/init.d/nginx reload "; # ?

logfile "/var/log/csync2.actions.log"; # ?

do-local; # , , , ,

}

auto younger;

}



The remarkable action item allows us to execute an arbitrary bash command after changes in certain files, in the specified example it will restart nginx.



I pay attention that the reload command, but not restart is executed. If, God forbid, you do not put a comma in the nginx config and it falls on a hundred machines, you will also quickly correct the error and on a hundred machines it will recover. In principle, it will be possible to change the config csync2, but this will take precious time downtime.

Conclusion

So, we got a very safe opportunity to edit the configs we are used to without any tricks, as if we had only one server, and change the changes to N other servers.

I note that csync2 does not have a head server, so changes can be made to any node and after running csync2 -x, changes will spill over to the others.



I synchronize via csync2 not only configs, but the actual project files themselves, some of which are uploaded by users, so I simply put csync2 in kroner (chase how many nodes I did? Right, on one and one time!

Lotions:

The other day I put the authorized_keys file in csync2 and now from my machine I go to any node without entering a password. It's great when passwords on all nodes are complex and different. You can also copy the hosts file to your computer and go to the nodes by their internal names.

Believe me, when you install, configure and debug a cluster of even seven, like my machines, it will save you a ton of time.

Little about the device

Running csync2 is very simple. The list of all files that you synchronize is stored in a local sqlite database and checked by the timestamp. It will not work to store hundreds of gigabytes of files there, but he scatters my 5-6 Gb scattered across several tens of thousands of files quite successfully. If you have more files, do not put csync2 in frequent crowns, it will start a bunch of copies and the servers will start to slow down. The base csync2 can be cleaned and generally carried out with it all sorts of manipulations. The program documentation (link below) also indicates the structure of the database used, so there is a lot of scope for creativity.

Analogs

There are systems that allow nodes to load remote configs. I didn't like them, but if you change cars in a cluster every day, you should look at them: Chef , Puppet .



============================

Useful links:

Csync2 documentation ( PDF )

Official csync2 website: http://oss.linbit.com/csync2/