Geekly Articles each Day
📜 ⬆️ ⬇️

Systemd logging control

Systemd journal

The systemd initialization daemon has already become a standard in modern Linux systems. Many popular distributions have switched to it: Debian, RHEL / CentOS, Ubuntu (since version 15.04). Systemd uses a fundamentally different (compared to the traditional syslog tool) logging approach.
It is based on centralization: the specialized journal component collects all system messages (kernel messages, various services and applications). In this case, there is no need to specifically configure sending logs: applications can simply write to stdout and stderr, a journal will save these messages automatically. Working in this mode is possible with Upstart, but it saves all the logs in a separate file, while systemd saves them in a binary database, which greatly simplifies the systematization and search.

Storing logs in binary files also avoids the difficulty of using parsers for different types of logs. If necessary, the logs can easily be converted to other formats (more details will be given below).
Journal can work both in conjunction with syslog, and completely replace it.
The journalctl utility is used to view logs. On the features and subtleties of working with her, we will discuss in this article.

')

Time setting



One of the major drawbacks of syslog is the preservation of records without regard to the time zone. In journal, this deficiency is eliminated: for logged events, you can specify both local time and coordinated universal time (UTC). Setting the time is done using the timedatectl utility.
You can view the list of time zones using the command:

 $ timedatectl list-timezones


Setting the desired time zone is as follows:

 $ timedatectl set-timezone <time zone>

After the installation is complete, it will be useful to make sure that everything is done correctly:

 $ timedatectl status
 Local time: Thu 2015-07-30 11:24:15 MSK
 Universal time: Thu 2015-07-30 08:24:15 UTC
 RTC time: Thu 2015-07-30 08:24:15
 Time zone: Europe / Moscow (MSK, +0300)
 NTP enabled: no
 NTP synchronized: no
 RTC in local TZ: no
 DST active: n / a


The very first line (Local time) should show the exact current time and date.

Journalctl: view logs



The journalctl utility is used to view logs.
If you enter the journaltl command without any arguments, a huge list will be displayed on the console:

 - Logs begin at Wed 2015-07-29 17:12:48 MSK, end at Thu 2015-07-30 11:24:15 MSK.  -
 Jul 29 17:12:48 host-10-13-37-10 systemd-journal [181]: Runtime journal is using 4.0M (max allowed 20.0M, trying to leave 30.0M free of 195.9M available → current limit 20.0M ).
 Jul 29 17:12:48 host-10-13-37-10 systemd-journal [181]: Runtime journal is using 4.0M (max allowed 20.0M, trying to leave 30.0M free of 195.9M available → current limit 20.0M ).
 Jul 29 17:12:48 host-10-13-37-10 kernel: Initializing cgroup subsys cpuset
 Jul 29 17:12:48 host-10-13-37-10 kernel: Initializing cgroup subsys cpu
 Jul 29 17:12:48 host-10-13-37-10 kernel: Initializing cgroup subsys cpuacct
 Jul 29 17:12:48 host-10-13-37-10 kernel: Linux version 3.16.0-4-amd64 (debian-kernel@lists.debian.org) (gcc version 4.8.4 (Debian 4.8.4- 1)) # 1 SMP Debian 3.16.7-ckt11-1 + deb8u2 (2015-07-17)
 Jul 29 17:12:48 host-10-13-37-10 kernel: Command line: BOOT_IMAGE = / boot / vmlinuz-3.16.0-4-amd64 root = UUID = b67ea972-1877-4c5b-a328-29fc0d6c7bc4 ro console = tty1 console = ttyS0 video = 640x480 consoleblank = 0 panic = 15 c


Here we have just a small fragment of it; in fact, it includes a huge number of entries.

Log filtering



The journalctl utility has options with which you can filter logs and quickly extract the necessary information from them.

View logs since the current download



With the -b option, you can view all the logs collected since the last system boot:

 $ journalctl -b


View logs of previous sessions



Using journalctl, you can view information about previous sessions of work in the system - in some cases this can be useful.
It should, however, be borne in mind that saving information about previous sessions is not supported by default in all Linux distributions. Sometimes it is required to activate

To do this, open the journald.conf configuration file, find the [Journal] section in it and change the value of the storage parameter to persistent:

 $ sudo nano /etc/systemd/journald.conf
 ...
 [Journal]
 Storage = persistent


You can view a list of previous downloads using the command:

 $ journalctl --list-boots

 0 9346310348bc4edea250555dc046b30c Thu 2015-07-30 12:39:49 MSK — Thu 2015-07-30 12:39:59 MSK


Its output consists of four columns. The first of them indicates the serial number of the download, the second - its ID, the third - the date and time. To view the log for a specific download, you can use identifiers from both the first and second columns:

 $ journalctl -b 0


or

 $ journalctl -b 9346310348bc4edea250555dc046b30c


Filter by date and time



The journalctl also has the ability to view logs for certain periods of time. To do this, use the options —since and –until. Suppose we need to view the logs starting from 17 hours 15 minutes July 20, 2015. To do this, you will need to run the command:
 $ journalctl --since "2015-07-20 17:15:00"


If no date is specified with the since option, logs will be output to the console starting from the current date. If the date is specified, but the time is not specified, the default time value “00:00:00” will be applied. Seconds are also optional (in this case, the default value is 00).

You can use the following human-readable constructions:

 $ journalctl --- since yesterday
 $ journalctl --since 09:00 --until now
 $ journalctl --since 10:00 --until "1 hour ago"


Filtering by application and service



To view the logs of a particular application or service, use the -u option, for example:

 $ journalctl -u nginx.service


The above command will log the nginx web server logs to the console.
Often there is a need to review the logs of a service for a certain period of time. This can be done with the help of the command:

 $ journalctl -u nginx.service --since yesterday


The -u option also uses date and time filtering, for example:

 $ journalctl -u nginx.service -u php-fpm.service —since today


This allows you to track the interaction of various services and to obtain information that could not be obtained when tracking the relevant processes separately.

Filter by process, user, and group



You can view logs for a process by entering its identification number (PID) on the journalctl command, for example:
 $ journalctl _PID = 381


To view the logs of processes running on behalf of a specific user or group, use _UID and _GID filters, respectively. Suppose we have a web server running on behalf of the user www-data. We first define the ID of this user:
 $ id -u www-data

 33

Now you can view the logs of all processes running on behalf of this user:

 $ journalctl _UID = 33


The list of users about which there are entries in the logs can be output to the console as follows:

 $ journalctl -F _UID


To view a similar list of user groups, use the command:

 $ journalctl -F _GUID


You can use other filters with the command journalctl. You can view a list of all available filters by running the command
 $ man systemd.journal-fields


Filtering along the way



You can also view logs for a process by specifying the path to it, for example:

 $ journalctl / usr / bin / docker


Sometimes this way you can get more detailed information (for example, view the records for all child processes).

View kernel messages



To view kernel messages, use the -k or −−dmesg option:

 $ journalctl -k


The following command will show all kernel messages for the current boot. To view kernel messages for previous sessions, use the -b option and specify one of the session identifiers (sequence number in the list or ID):

 $ journalctl -k -b -2


Error message filtering



While diagnosing and fixing problems in the system, it is often necessary to review the logs and find out if there are any critical error messages in them. Especially for this journalctl provides the ability to filter by error level. You can view messages about all errors that occurred in the system using the -p option:

 $ journalctl -p err -b


The above command will show all error messages that occurred in the system.

These messages can be filtered by level. The journal uses the same classification of error levels as syslog:



Error level codes are specified after the -p option.

Logging to standard output



By default, journalctl uses the external utility less to display log messages. In this case, it is impossible to apply standard utilities for processing text data (for example, grep) to them. This problem is easily solved: just use the −−no-pager option, and all messages will be written to the standard output:

 $ journalctl --no-pager


After that they can be transferred to other utilities for further processing or saved in a text file.

Select output format



Using the -o option, you can convert the log data into various formats, which makes it easier to parse and further processing, for example:

 $ journalctl -u nginx.service -o json

 { "__CURSOR": "s = 13a21661cf4948289c63075db6c25c00; i = 116f1; b = 81b58db8fd9046ab9f847ddb82a2fa2d; m = 19f0daa; t = 50e33c33587ae; x = e307daadb4858635", "__REALTIME_TIMESTAMP": "1422990364739502", "__MONOTONIC_TIMESTAMP": "27200938", "_BOOT_ID" "81b58db8fd9046ab9f847ddb82a2fa2d", "PRIORITY": "6", "_UID": "0", "_GID": "0", _ _:::::::::::::::::::,,,,,: c c desktop "," SYSLOG_FACILITY ":" 3 "," CODE_FILE ":" src / core / unit.c "," CODE_LINE ":" 1402 "," CODE_FUNCTION ":" unit_status_log_starting_stopping_reloading "," SYSLOG_IDENTIFIER ":" d d d d ing _ _ pping pping re:: "system system" "d art art SY SY SY SY SY SY SY SY art art art SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY SY" SYSLOG_FACILITY " MESSAGE_ID ":" 7d4958e842da4a758f6c1cdc7b36dcc5 "," _TRANSPORT ":" journal "," _PID ":" 1 "," _COMM ":" systemd "," _EXE ":" / usr / lib / systemd / systemd "," _CMDLINE ":" / usr / lib / systemd / systemd "," _CMDLLE "_CMM": " "/ usr / lib / systemd / systemd", "_SYSTEMD_CGROUP": "/", "UNIT": "nginx.service", "MESSAGE": "Starting a high performance web server and reverse proxy server ...", "_SOURCE_REALTIME_TIMESTAMP": "1422990364737973"}


The json object can be represented in a more structured and human-readable form, specifying the format json-pretty or json-sse:

 $ journalctl -u nginx.service -o json-pretty

 {
     "__CURSOR": "s = 13a21661cf4948289c63075db6c25c00; i = 116f1; b = 81b58db8fd9046ab9f847ddb82a2fa2d; m = 19f0daa; t = 50e33c33587ae; x = e307daadb4858635",
     "__REALTIME_TIMESTAMP": "1422990364739502",
     "__MONOTONIC_TIMESTAMP": "27200938",
     "_BOOT_ID": "81b58db8fd9046ab9f847ddb82a2fa2d",
     "PRIORITY": "6",
     "_UID": "0",
     "_GID": "0",
     "_CAP_EFFECTIVE": "3fffffffff",
     "_MACHINE_ID": "752737531a9d1a9c1e3cb52a4ab967ee",
     "_HOSTNAME": "desktop",
     "SYSLOG_FACILITY": "3",
     "CODE_FILE": "src / core / unit.c",
     "CODE_LINE": "1402",
     "CODE_FUNCTION": "unit_status_log_starting_stopping_reloading",
     "SYSLOG_IDENTIFIER": "systemd",
     "MESSAGE_ID": "7d4958e842da4a758f6c1cdc7b36dcc5",
     "_TRANSPORT": "journal",
     "_PID": "1",
     "_COMM": "systemd",
     "_EXE": "/ usr / lib / systemd / systemd",
     "_CMDLINE": "/ usr / lib / systemd / systemd",
     "_SYSTEMD_CGROUP": "/",
     "UNIT": "nginx.service",
     "MESSAGE": "Starting A high performance web server and reverse proxy server ...",
     "_SOURCE_REALTIME_TIMESTAMP": "1422990364737973"
 }


In addition to JSON, log data can be converted to the following formats:



View information about recent events



The option -n is used to view information about recent events in the system:

 $ journalctl -n


By default, the console displays information about the last 10 events. With the -n option, you can specify the required number of events:

 $ journalctl -n 20


View logs in real time



Messages from logs can be viewed not only in the form of saved files, but also in real time. To do this, use the -f option:

 $ journalctl -f


Logging control



Determining the current log volume



Over time, the volume of logs grows, and they take up more and more space on the hard disk. You can find out the amount of currently available logs using the command:

 $ journalctl --disk-usage
 Journals take up 16.0M on disk.


Log rotation



Setting the rotation of the logs is carried out using the options −− vacuum-size and −− vacuum-time.
The first of these sets the maximum allowable size for logs stored on disk (in our example, 1 GB):

 $ sudo journalctl --vacuum-size = 1G


As soon as the volume of logs exceeds the specified figure, the extra files will be automatically deleted.
The −−acacuum-time option works in a similar way. It sets a retention period for logs, after which they will be automatically deleted:

 $ sudo journalctl --vacuum-time = 1years


Setting the rotation logs in the configuration file



Log rotation settings can also be specified in the /tc/systemd/journald.conf configuration file, which includes the following parameters, among others:



Central storage of logs



One of the most common tasks in the work of the system administrator is to configure the collection of logs from several machines and then place them in a centralized storage.
Systemd has special components for this task: systemd-journal-remote , systemd-journal-upload and systemd-journal-gatewayd .

Using the systemd-journal-remote command, you can receive logs from remote hosts and save them (the systemd-journal-gatewayd daemon must be running on each of these hosts), for example:

 $ systemd-journal-remote −−url https://some.host:19531/


As a result of executing the above command, logs from the host some.host will be saved in the var / log / journal / some.host / remote-some ~ host.journal directory.

Using the systemd-journal-remote command, you can also add the logs on the local machine to a separate directory, for example:

 $ journalctl -o export |  systemd-journal-remote -o / tmp / dir -


The systemd-journal-upload command is used to upload logs from a local machine to a remote repository:

 $ systemd-journal-upload --url https://some.host:19531/


As can be seen from the above examples, the systemd native utilities for supporting centralized logging are simple and easy to use. But, unfortunately, they are far from being included in all distributions, but only in Fedora and ArchLinux.

Users of other distributions so far have to send logs to syslog or rsyslog, which then send them over the network. Another solution to the problem of centralized logging was suggested by the developers of the journal2gelf utility included in the systemd official repository: the journaltl output in JSON format is converted to GELF format and then transferred to the application for collecting and analyzing Graylog logs. The solution is not very convenient, but nothing better can be invented in the current situation. It remains only to wait for the "native" components to be added to all distributions.

Conclusion



As can be seen from the review done, the systemd journal is a flexible and convenient tool for collecting system and application data. A high level of flexibility and convenience was achieved, firstly, due to the centralized approach to logging, and secondly, thanks to the automatic writing of detailed metadata to the logs. Using journalctl, you can view logs, getting the necessary information to analyze the work and debug various system components.
If you have questions and additions - welcome to comments. The discussion about systemd and its components will be continued in the following publications.

Readers who for one reason or another can not leave comments here are invited to our blog .

Source: https://habr.com/ru/post/264731/

More articles:


All Articles