Audit of system events in Linux

One of the tools to improve the security level in Linux is the audit subsystem. With its help you can get detailed information about all system events.
It does not provide any additional protection, but provides detailed information on security breaches, based on which specific measures can be taken. The features of working with the audit subsystem will be discussed in this article.

Audit subsystem: architecture and operation

The audit subsystem has been added to the Linux kernel since version 2.6. It is designed to keep track of security-critical system events. Examples of such events include the following (the list is far from complete):

• system startup and shutdown;
• reading, writing and changing file permissions;
• initiating network connections;
• attempts of unsuccessful authorization in the system;
• change network settings;
• changing user and group information;
• start and stop applications;
• make system calls.

None of the named events can occur without the use of kernel system calls. To track them, simply intercept the corresponding system calls. This is what the audit subsystem does:
')


Having received a call from an application in user space, the audit subsystem passes it through one of the following filters: user, task, or exit (for more details, see below). After that, the call is passed through the exclude filter, which, based on the audit rules, passes it to the auditd daemon for further processing.

Such a simple scheme makes it possible to effectively monitor any aspect of the OS, and in the event of a compromise of the system, identify suspicious actions and determine their cause.

Installation

To start working with the audit subsystem, you need to install the auditd package (hereinafter, examples of commands for OC Ubuntu 14.04 are provided):

 $ sudo apt-get install auditd

This package includes the auditd daemon and several auxiliary utilities:

• auditctl - utility to control the auditd daemon; allows you to receive information about the current state of the audit subsystem, as well as add and delete rules;
• autrace - a utility for auditing events generated by processes (works on the same principle as strace);
• ausearch is a utility for searching events in log files;
• aureport is a utility for generating reports on the operation of the audit system.

Configuration

Audit settings are stored in the configuration file etc / audit / auditd.conf. It contains, among others, the following parameters:

• log_file - the file in which the logs of the audit subsystem will be stored;
• log_format - the format in which logs will be saved;
• freq is the maximum number of protocol records that can be stored in the buffer;
• flush - buffer synchronization with disk (none - do nothing, incremental - transfer data from buffer to disk at the frequency specified in the value of the freq parameter; data - synchronize immediately, sync - synchronize both data and file metadata when writing to disk );
• max_log_file - maximum log file size in megabytes;
• max_log_file_action - action when the maximum size of the log file is exceeded;
• space_left - minimum free space in megabytes, after which the action specified in the following parameter should be implemented;
• space_left_admin - indicates what to do when there is not enough free space on the disk (ignore - do nothing; syslog - send to syslog, email - send notification by mail; suspend - stop writing logs to disk; single - go to single-user mode; halt - turn off the car)
• disk_full_action is the action to be taken when the disk is full (this parameter can take the same values ​​as space_left_admin).

Creating rules

To add and configure rules, use the auditctl command. Here is a list of its options:

• -l - list the existing rules;
• -a - add a new rule;
• -d - remove the rule from the list;
• -D - delete all existing rules.

To create a new rule, you need to run a command like:

	
 $ auditctl -a <list>, <action> -S <name of system call> -F <filters>

First, after the -a option, the list is added to which the rule should be added. There are 5 such lists in total:

• task - events related to the creation of new processes;
• entry - events that occur when entering a system call;
• exit - events that occur when leaving the system call;
• user - events using user space parameters;
• exclude - used to exclude events.

It then indicates what to do after the event. There are two possible options: always (events will be recorded in the log) and never (will not).

After the -S option comes the name of the system call, at which the event needs to be intercepted (open, close, etc.).

After the -F option, additional filtering options are specified. For example, if we need to audit the access to files from the / etc directory, the rule would look like this:

 $ auditctl -a exit, always -S open -F path = / etc /

You can install an additional filter:

 $ auditctl -a exit, always -S open -F path = / etc / -F perm = aw

The abbreviation aw means the following: a - attribute change (attribute change), w - write (write). The wording perm = aw indicates that for the / etc directory you need to keep track of all facts of attribute changes (a - attribute change) and w (w - write).

When setting up tracking for individual files, you can omit the -S option, for example:

 $ auditctl -a exit, always -F path = / etc / -F perm = aw

Rule files

The rules can not only be set via the command line, but also written in the file etc / audit / audit.rules.
This file begins with the so-called meta-rules, in which general logging settings are specified:

 # delete all previously created rules
 -D

 # set the number of buffers in which messages will be stored
 -b 320

 # specify what to do in a critical situation (for example, when buffers overflow): 
 0 - do nothing;  1 - send the message to dmesg, 2 - send the kernel to panic
 -f 1 

This is followed by custom rules. Their syntax is very simple: just list the corresponding options of the auditctl command. Consider an example of a type rule file:

 # track unlink () and rmdir () system calls
 -a exit, always -S unlink -S rmdir

 # track open () system calls from user with UID 1001
 -a exit, always -S open -F loginuid = 1001

 # track access to password and group files and attempt to change them:
 -w / etc / group -p wa
 -w / etc / passwd -p wa
 -w / etc / shadow -p wa
 -w / etc / sudoers -p wa

 # track access to the following directory:
 -w / etc / my_directory -pr

 # close access to configuration file to prevent changes
 -e 2

Configuration changes will take effect after restarting the auditd daemon:

 $ sudo service auditd restart

Log file analysis: aureport utility

All log files are stored in the / var / log / audit directory in machine-readable format. They can be made intelligible with the aureport utility.

If you enter the aureport command without arguments, we will see general system statistics (number of system users, total number of system calls, number of open terminals, etc.):

 $ sudo aureport

 Summary Report
 ======================
 Range of time in logs: 07/31/2015 14: 04: 23.870 - 08/04/2015 09: 37: 13.200
 Selected time for report: 07/31/2015 14:04:23 - 08/04/2015 09: 37: 13.200
 Number of changes in configuration: 0
 Number of changes to accounts, groups, or roles: 3
 Number of logins: 0
 Number of failed logins: 0
 Number of authentications: 0
 Number of failed authentications: 61205
 Number of users: 2
 Number of terminals: 5
 Number of host names: 73
 Number of executables: 6
 Number of files: 0
 Number of AVC's: 0
 Number of MAC events: 0
 Number of failed syscalls: 0
 Number of anomaly events: 0
 Anomaly events: 0
 Number of crypto events: 0
 Number of keys: 0
 Number of process IDs: 17858
 Number of events: 61870

It does not have much practical value. The specialized reports are of much greater interest. So, for example, you can view information about all system calls:

 $ sudo aureport -s

 Syscall report
 =======================================
 # date time syscall pid comm auid event
 =======================================
 1. 08/03/2015 15:45:03 313 10285 modprobe -1 52501
 2. 03/08/2015 15:45:03 313 10290 modprobe -1 52502
 3. 08/03/2015 15:45:03 54 10296 iptables -1 52503
 4. 08/03/2015 15:45:03 54 10302 iptables -1 52504
 5. 08/03/2015 15:45:03 54 10305 iptables -1 52505
 6. 03/08/2015 15:45:03 54 10313 iptables -1 52506
 7. 03/08/2015 15:45:03 54 10325 iptables -1 52507
 8. 03/08/2015 15:45:03 54 10329 iptables -1 52508
 9. 03/08/2015 15:45:03 54 10343 iptables -1 52509
 10.08 / 03/2015 15:45:03 54 10345 iptables -1 52510
 11.08 / 03/2015 15:45:03 54 10349 iptables -1 52511

Using the -au (or −−auth) option, you can view information on all login attempts:

 $ sudo aureport -au 

 Authentication Report
 ============================================
 # date time acct host term exe success event
 ============================================
 1. 08/31/2015 18:00:19 ubnt static-166-6-249-80.stalowa.pilicka.pl ssh / usr / sbin / sshd no 333
 2. 08/31/2015 18:01:38 root 59.63.188.31 ssh / usr / sbin / sshd no 334
 3. 08/31/2015 18:01:41 root 59.63.188.31 ssh / usr / sbin / sshd no 335
 4. 08/31/2015 18:01:45 root 59.63.188.31 ssh / usr / sbin / sshd no 336
 5. 08/31/2015 18:01:53 root 59.63.188.31 ssh / usr / sbin / sshd no 337
 6. 08/31/2015 18:01:57 root 59.63.188.31 ssh / usr / sbin / sshd no 338
 7. 08/31/2015 18:01:59 root 59.63.188.31 ssh / usr / sbin / sshd no 339

Aureport supports filtering by date and time:

 $ sudo aureport -s --start 07/31/15 12:00 --end 07/31/15 13:00

You can specify both a specific time and date, and special human-readable constructions:

• now - the current time;
• yesterday is the day before;
• recent - 10 minutes ago;
• this-week (or this-month, this-year) is the current week (month, year).

With aureport, you can view information about the actions of any user of the system. To do this, you must first find out the id of this user:

 $ id user
 uid = 1000 (user) gid = 1000 (andrei) groups = 1000 (andrei), 27 (sudo)

and then run the following command:

 $ sudo ausearch -ui 1000 --interpret

Ausearch: search and analysis of events

To view detailed information about the event, use the ausearch utility:

 $ sudo ausearch -a <event number> 

The output from the above command looks like this:

type=SYSCALL msg=audit(1364481363.243:24287): arch=c000003e syscall=2 success=no exit=-13 a0=7fffd19c5592 a1=0 a2=7fffd19c4b50 a3=a items=1 ppid=2686 pid=3538 auid=500 uid=500 gid=500 euid=500 suid=500 fsuid=500 egid=500 sgid=500 fsgid=500 tty=pts0 ses=1 comm="cat" exe="/bin/cat" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key="sshd_config"

Consider its structure in more detail. The type field indicates the type of the record; type = syscall means that the entry was made after the system call was made. The msg field shows the time of the event in the Unix Timestamp format and its unique identification number.

The arch field contains information about the system architecture used (c000003e means x86_84), presented in a coded hexadecimal format. To display it in a human-readable form, you can use the -i or –−interpret option.

In the syscall field, the type of system call is specified - in our case it is 2, that is, the call to open. The success parameter indicates whether the call was processed successfully or not. In our example, the call was not processed successfully (success = no).

For each call, the report also lists individual parameters; You can read more about them in the official manual . Information on any parameter in human-readable form can be output to the console using the -i or −−interpret option mentioned above, for example:

 $ sudo ausearch --interpet --exit -13

The -sc option allows listing events related to the specified system call, for example:

 $ sudo ausearch -sc ptrace

The option -ui is used to search for events by user ID:

 $ ausearch -ui 33

Search for daemon names using the -tm option:

 $ ausearch -x -tm cron

You can also use keys to search for desired events, for example:

 $ sudo auditctl -k root-actions

The above command will list all actions performed on behalf of the root user. Filtering by date and time is also supported, similar to the one described above. You can display a list of failed events using the –−failed option.

Analyzing processes using the autrace utility

In some cases, it may be useful to obtain information about events related to one particular process. For this purpose you can use the autrace utility. Suppose we need to track the date process and find out which system calls and files it uses. Run the command:

 $ sudo autrace / bin / date

The following text will appear on the console:

 Waiting to execute: / bin / date
 Mon Aug 31 17:06:32 MSK 2015
 Cleaning up ...
 Trace complete.  You can locate the records with 'ausearch -i -p 29234'

Let's pay attention to the last line of the output: it contains a command with which you can get more detailed information. Run this command and pass the output to the aureport utility, which converts it into a human-readable format:

 $ sudo ausearch -p 29215 --raw |  aureport -f -i

As a result, we get this report:

 File report
 ===============================================
 # date time file syscall success exe auid event
 ===============================================
 1. 08/31/2015 16:52:16 / bin / date execve yes / bin / date root 25
 2. 08/31/2015 16:52:16 /etc/ld.so.nohwcap access no / bin / date root 27
 3. 08/31/2015 16:52:16 /etc/ld.so.preload access no / bin / date root 29
 4. 08/31/2015 16:52:16 /etc/ld.so.cache open yes / bin / date root 30
 5. 08/31/2015 16:52:16 /etc/ld.so.nohwcap access no / bin / date root 34
 6. 08/31/2015 16:52:16 /lib/x86_64-linux-gnu/libc.so.6 open yes / bin / date root 35
 7. 08/31/2015 16:52:16 / usr / lib / locale / locale-archive open yes / bin / date root 52
 8. 08/31/2015 16:52:16 / etc / localtime open yes / bin / date root 56

Central storage of logs

To send the audit subsystem logs to the centralized repository, use the audisp-remote plugin. It is included in the audisp-plugins package, which must be additionally installed:

 $ sudo apt-get install audisp-plugins

The configuration files for all plugins are stored in the /etc/audisp/plugins.d directory.

Remote logging settings are specified in the /etc/audisp/plugins.d/audisp-remote.conf configuration file. By default, this file looks like this:

 active = no
 direction = out
 path = / sbin / audisp-remote
 type = always
 #args =
 format = string

To activate sending logs to the remote storage, replace the value of the active parameter with yes. Then open the file etc / audisp / audisp-remote.conf and as the value of the remote_server parameter specify the letter or IP address of the server where the logs will be stored.

To receive logs from remote hosts and store them on the server, the following parameters need to be written in the /etc/audit/auditd.conf file:

tcp_listen_port = 60
tcp_listen_queue = 5
tcp_max_per_addr = 1
##tcp_client_ports = 1024-65535 #optional
tcp_client_max_idle = 0


Conclusion

In this article, we outlined the basics of working with the Linux audit subsystem. We reviewed the principle of the audit system, learned how to formulate rules, read logs and use auxiliary utilities.

For those who want to explore the topic in more detail here are some useful links:

• full list of types of audit records ;
• full list of possible events ;
• article on the use of audit capabilities in the investigation of attacks on the kernel .

Readers who are not able to leave comments here are invited to our blog .