Remote logging in journald or Still “you don't need this”?

Disclaimer

All experiments were performed on CentOS Linux release 7.2.1511 as the main system, with the latest available from the systemd turnip reps (systemd-219-19.el7_2.13). I hope some of the data will be irrelevant at the time of publication of the article.

Prologue

Starting to capture linux distros with the release of Fedora 15, systemd finally won. Bison and aksakals are gradually accustomed to units and systemctl. The last defenders of the Good Old are gnashing their teeth. In these realities, it is impossible to bypass the systemd child products. And today, let's talk, for example, about journald.

By itself, journald (and, accordingly, journalctl) is a great tool. Originated as a slightly third-party systemd project, by now journald has become the second tool from the systemd family that system administrators are familiar with. I really like the idea of ​​running log storage with an optional ability to reset to permanent storage, an idea with journalctl --boot boot boot-id and machine-id ( journalctl --machine ), the ability to call logs of any registered journalctl -u applications from a single interface, and the presence of out-of-the-box log rotation (both in place and in time) using journalctl --vacuum-size or journalctl --vacuum-time . From the good one can not fail to mention also the parameters since , until and priority , which can “fade” events in time and priority, and, of course, the utc parameter, which removes a huge headache with multi-timezone teams and projects. The binary file storage format may be a bit controversial, but this choice was explained by the authors even at the first journald announcements (safety and cheap storage, integration with systemd, forced uniform format, portability).

Unfortunately, the journald directory mechanism, which allows, for example, organizing the translation of log messages or providing urls to end-users for solving problems with specific errors, has not acquired much of the distribution. But in general, even the systemd developers do not take full advantage of this opportunity - to us, mortals.

This is all implemented, works, described in hundreds of manuals, tested. Thank you very much, but I want more.

Preamble

Not long ago, a friend asked him to set up a server for collecting logs. Hha! - I thought, - this is a great reason to study journald in the light of its capabilities of the central log server!

It is clear that the tool depends on the task. So this time was a brief collection of requirements:

1. It is necessary to organize a single point of logging.
2. It is necessary to organize the ability to connect to a single point of logging in advance of an unknown number of clients (and disconnect them, respectively)
3. It is advisable to organize the ability to view logs online, with automatic receipt of new records.
4. During the previous step, the storage of logs at the central point can be limited to N hours (not even days)
5. The message format is inconsistent down to the multiline.
6. The key software of interest is written to stdout (yes, you can change the behavior or nakostilyt, but buy what they sell), the console does not close

And general technical introductory:

1. Data is transmitted over public networks.
2. Head dedicated machine: Centos7.2 (latest)
3. Connected machines: Ubuntu 16.04

It seems that systemd and journald are a good choice, right? After all, all the items have already been implemented! Raise the test stand!

Host host

Well, let's start.
We are interested in three components:

1. systemd-journal-gatewayd - http-daemon, opening the port to view (or download) journal entries
2. systemd-journal-remote - a daemon that downloads or accepts journal entries on a central server.
3. systemd-journal-upload - daemon duplicating journal entries to a remote server

All these components are included in the systemd-journal-gateway centos package, so we execute:

 yum -y update && yum -y install systemd-journal-gateway 

The systemd-journal-remote daemon is used to receive logs on the host machine. From him and begin.

systemd-journal-remote

Remote has two modes of operation: active (in which it goes to the remote log itself and downloads logs - including in tracking mode. For this mode, all remote machines need systemd-journal-gatewayd ) and passive (in which the daemon hangs and waiting for him to come). Taking into account the unknown number of machines - choose the passive mode. The whole configuration, in fact, is to make the /var/log/journal/remote directory, assign the rights of the desired user to it and start the service:

 mkdir -p /var/log/journal/remote chown systemd-journal-remote:systemd-journal-remote /var/log/journal/remote systemctl start /var/log/journal/remote 

Since we are collecting a demo booth, let's switch the file reception protocol from https to http. To do this, you need to edit the starting service file /lib/systemd/system/systemd-journal-remote.service , replacing the option listen-https with listen-http . It will also be useful to correct /lib/systemd/system/systemd-journal-remote.socket , specifying only the address of interest for binding the daemon.

The daemon started, hangs in the passive mode, works on http. Hooray!

The devil is in the details

First, when creating the /var/log/journal directory, all your system logs will be written to the /var/log/journal/<uuid> directory. To change this behavior and return as it was, you need to fill in the configuration file /etc/systemd/journald.conf (or even better /etc/systemd/journald.conf.d/< >.conf , because the first one can be wiped with updates), namely the string Storage= . By default, the value of this parameter is auto , which means "if there is a directory in var, journald writes logs there." In my case, the parameter had to be forced to set to volatile .

systemd-journal-upload

Upload is still easier: create /etc/systemd/journal-upload.d/< >.conf , writing there:

 [Upload] URL=http://<IP- remote>:<  > 

and run systemctl start systemd-journal-upload

The devil is in the details

Secondly, the demon will not start due to a very old bug in Centos . So do usermod with your hands: usermod -a -G systemd-journal systemd-journal-upload . After that, the daemon should start successfully.

Remote clients

On remote clients, I remind you, it costs Ubuntu 16.04. So we put apt-get install systemd-journal-remote and make /etc/systemd/journal-upload.d/< >.conf changes /etc/systemd/journal-upload.d/< >.conf , similar to the previous paragraph (except usermod ).

systemd-journal-gatewayd

And here, in fact, nothing to describe. The current version presented in Centos does not allow specifying non-standard directories for the specified daemon. Moreover, the standard directory for logs from other machines, according to the logic of developers, is non-standard for the log viewer. In the new versions of systemd, this seems to be fixed , but we will not install non-native versions ...

Well, let's at least watch the logs:

 journalctl -D /var/log/journal/remote --follow 

Well, something like it works ...

The devil is in the details

Thirdly, thanks to the old bug in systemd , the directory /var/log/journal/remote will start to swell in the wildest. And then nothing will help you ...

Let's get to the end!

However, since we have come this far, let's get to the end. Run the systemd-journal-gatewayd daemon on the host machine (again, do not forget to fix /lib/systemd/system/systemd-journal-gatewayd.socket again to limit the daemon, right?) And carefully examine the web-browser viewer:

• Online logging is virtually absent
• Fierce on mouse over and on mouse scroll for logs
• microhttpd under the hood
• Sorry, but terrifying interface
• It is possible to filter logs by systemd-unit, but (due to the number of non-informative session-units for example from cron-scripts) the inability to use filters
• All units are written completely - if you use for example passwords to databases in cron strings, your passwords will wait for you

NDA Perhaps it’s good that it wasn’t possible to set it up, m?

Total

For the proof of concept, a pet-project was written in a couple of days, the blessing of journald provides a native library for python. Of the features:

1. Online browsing by SSE
2. Ability to filter output units by the administrator at the configuration level
3. Ability to filter hosts / priority / units by the user at the web application level
4. Well, a little bootstrap

The same project was unrolled to a friend, with a description of the problems, a disclaimer and a forced cleaning of magazines.

But on large projects personally, I probably will not use journald as a central repository of logs for a long time. Until the above bugs are removed, my choice is definitely in favor of syslog.

Article author: Stepan Karamyshev