Logreplica: collecting logs from the entire cluster into a single point in real time

I continue to share useful utilities that I use in various projects. This time it will be about logreplica - a simple tool that allows you to organize a reliable transfer of logs from different servers of the cluster to a single machine with large disks “in real time”. This is very convenient if you want to monitor or analyze logs from the entire cluster centrally, as if they were written directly to a single machine.



We can say that logreplica was conceived as a more convenient and reliable way to collect logs in a central place, rather than using the syslog / syslog-ng settings.



The advantage of logreplica is in its simplicity of configuration: you only set up the “mask” of the names of the log files and specify the addresses of the source machines, and later the logs corresponding to the mask automatically and on the fly add up to the central machine (including - Sources of new log files appear, unknown at the time of launch logreplica). When adding a new machine on it, you do not need to tweak anything: it is enough to include its name in the config file.

')

Config example /etc/dklab_logreplica.conf

 # What is the directory logs to?
 destination = / var / log / cluster

 # Skip these log files prefixes while storing to destination directory.
 skip_destination_prefixes = / var / log: / var / lib / pgsql / data / logs

 # Path to internal state directory (may not change).
 scoreboard = /var/run/dklab_logreplica.scoreboard

 # Time interval to log growth checks.
 delay = 0.25

 # Default user to connect to remote hosts.
 user = root

 # Files
 [files]
 / var / log / {messages, maillog}
 / var / log / httpd / * _ log

 # What hosts to connect to gather logs from.
 [hosts]
 first = machine1.example.com
 second=nobody@machine2.example.com

How to install and configure logreplica

On source machines, you do not need to configure anything: they immediately appear in the system after being written into the central config file. So, several dozens of servers can be freely connected to logreplica without configuration: all settings are located in a single /etc/dklab_logreplica.conf file.

1. Select the server on which all logs will be added; copy the logreplica distribution to / opt / dklab_logreplica / .
2. Copy the dklab_logreplica.init init script to /etc/init.d and configure its autorun when the machine boots (see chkconfig , update-rc.d and other Linux utilities).
3. Copy the default dklab_log replica.conf.sample config to /etc/dklab_logreplica.conf and change it according to your system:
   • set the machines from which logs will be collected;
   • specify the mask names of the log files to be monitored;
   • specify the directory where logs will be copied in real time.
4. Create a private + public key using ssh-keygen -t rsa . Spread the public key on the machines from which the logs will be downloaded: ssh-copy-id root @ machine-to-be-pulled . As a result, you should get password-free access from the log server to all source machines, otherwise logreplica will not work.
   
   
5. Now run /etc/init.d/dklab_logreplica start - and logreplica will start tracking changes in the log files on the source machines and add the data to the destination directory.

The logreplica daemon can reestablish communication with source machines if it suddenly disappears due to network problems. In this case, all data that has accumulated in the logs during idle time will be transferred, so nothing will be lost. Also logreplica tracks the rotation of the logs on the source machines and correctly processes it.

Problems solved by logreplica

Suppose there are several physical or virtual machines in the cluster that perform various tasks (for example, SQL server, web-frontend, balancer, mail server, etc.). Sometimes you want to have logs from all these machines in a single place - for example, to read various statistics or simply conveniently (even if using tail -f) to monitor what is happening in the entire system.



Of course, you can configure syslog or syslog-ng to transfer all logs over the network to the central log server, however, if you do this, you will encounter a number of inconveniences:

1. In the case of temporary interruptions of communication between machines (this happens, unfortunately, more often than we would like) pieces of logs can be lost, so you will sometimes lose data.
2. In practice, it is rather difficult to maintain the syslog or syslog-ng configuration synchronized with the real state of affairs in the cluster: new machines can be added, a set of log files can be changed, and so on.
3. Not all services support logging to syslog (for example, apache and nginx write them directly to files to minimize latency). So you have to use named pipes to transfer data to syslog, which means configs and dependencies will grow and grow.
4. Often you want to replicate the log files to the central machine according to the mask, rather than rigidly specifying the name of the log file. Syslog and syslog-ng cannot do this.
5. Finally, in practice, it is very convenient to store logs not only on the central machine, but also on the machine where they are created (a narrower “rotation window” can be configured on the source machine). So your syslog and syslog-ng configs grow again ...

The logreplica utility was created in order to solve all these problems. In general, logs are simply “magically” and in real-time on a central machine, without any changes in the settings of services and remote machines.



Links: logreplica on dklab.ru , logreplica on GitHub . Write in the comments if you know analogs or have a different vision of the problem described.