Monitoring Information Security Events with ZABBIX

When I first saw Zabbix, I thought, why not try using it as a solution for monitoring information security events. As you know, in the IT infrastructure of a company there are a lot of various systems that generate such a flow of information security events that it is simply impossible to view them all. Now in our corporate monitoring system there are hundreds of services that we observe with a high degree of detail. In this article, I consider the features of using Zabbix as a solution for monitoring information security events.

What allows Zabbix to solve our problem? Something like this:

• Maximum automation of resource inventory, vulnerability management, compliance control with security policies and changes.
• Real-time protection of corporate resources using automatic information security monitoring.
• The ability to get the most reliable picture of network security.
• Analysis of a wide range of complex systems: network equipment such as Cisco, Juniper, Windows platforms, Linux, Unix, MSQL DBMS, Oracle, MySQL, etc., network applications and web services.
• Minimize audit costs and control security.

In the article I will not consider all of the above, we will only address the most common and simple questions.

Training

So, for starters, I installed the Zabbix monitoring server. As a platform, we will use the FreeBSD OS. I think that there is no need to tell in details about the installation and configuration process, rather detailed documentation in Russian is available on the developer’s website, starting from the installation process to describing all the system’s capabilities.
We will assume that the server is installed, configured, and the web-frontend is configured to work with it. At the time of writing, the system is running FreeBSD 9.1, Zabbix 2.2.1.

Monitoring MS Windows Server Security Events

With the help of the Zabbix monitoring system, you can collect any available information from the Windows system logs with an arbitrary level of detail. This means that if Windows writes an event to the log, Zabbix “sees” it, for example, by Event ID, text or binary mask. In addition, using Zabbix, we can see and collect a huge number of events that are interesting for monitoring security, for example: running processes, open connections, drivers loaded into the kernel, used dll, logged in via console or remote user access, and much more.
')
All that remains is to determine the events arising from the realization of the threats we expect.

Establishing a solution for monitoring IS events in the IT infrastructure, one should take into account the need to choose a balance between the desire to monitor everything and the possibilities of processing a huge amount of information on IS events. Here Zabbix opens up great opportunities for choice. Zabbix key modules are written in C / C ++, the speed of recording from the network and processing of monitored events is 10 thousand new values ​​per second on a more or less ordinary server with a properly configured database system.

All this gives us the opportunity to track the most important security events on the observed network node running Windows.

So, for a start, let's look at a table with an Event ID, which, in my opinion, can obviously be used to monitor information security events:

IS events MS Windows Server Security Log

EventID Description	2008 Server	2003 Server
Clearing the audit log	1102	517
Signed in successfully	4624	528, 540
Account failed to log in	4625	529-535, 539
Created user account	4720	624
Attempt to reset account password	4724	628
User account disabled	4725	629
User account removed	4726	630
Created a secure local security group	4731	635
Added member to secure local group	4732	636
Removed member from protected local group	4733	637
Removed secure local security group	4734	638
Changed secure local security group	4735	639
User account changed	4738	642
User account locked	4740	644
Account name has been changed	4781	685

I pay attention to local security groups, but in more complex AD schemes, it is also necessary to consider common and global groups.
In order not to duplicate information, more information about critical events can be found in the article:
http://habrahabr.ru/company/netwrix/blog/148501/

Methods for monitoring IS events MS Windows Server

Consider the practical application of this problem.
To collect data you need to create a new data item:

: eventlog[Security,,,,1102|4624|4625|4720|4724|4725|4726|4731|4732|4733|4734|4735|4738|4781]   : Zabbix  ()  :  () 

If desired, for each Event ID you can create a separate data item, but I use several Event IDs in one key to store all the records in one place, which allows you to quickly search for the necessary information without switching between different data items.

I want to note that in this key we use the Security event log as the name.
Now that we have the item, we need to set up a trigger. A trigger is a Zabbix mechanism that allows you to signal that any of the monitored events has occurred. In our case, this is an event from the server log or MS Windows workstation.

Now all that will record the audit log with the specified Event ID will be transmitted to the monitoring server. Specifying a specific Event ID is useful because we receive only the necessary information, and nothing superfluous.

Here is one of the trigger expressions:

 {Template Windows - Eventlog 2008:eventlog[Security,,,,1102|4624|4625|4720|4724|4725|4726|4731|4732|4733|4734|4735|4738|4781].logeventid(4624)}=1&{Template Windows - Eventlog 2008:eventlog[Security,,,,1102|4624|4625|4720|4724|4725|4726|4731|4732|4733|4734|4735|4738|4781].nodata(5m)}=0 

This expression will allow displaying on the Dashboard information that “Login with a user account was successful”, which corresponds to Event ID 4624 for MS Windows Server 2008. The event will disappear after 5 minutes, if during this time you have not logged in again.

If you need to track a specific user, for example “Administrator”, you can add a regexp check to the trigger expression:

 &{Template Windows - Eventlog 2008:eventlog[Security,,,,1102|4624|4625|4720|4724|4725|4726|4731|4732|4733|4734|4735|4738|4781,,skip].regexp()}=1 

Then the trigger will work only if it is logged in to the system under an account named “Administrator”.

PS

We considered the simplest example, but you can also use more complex structures. For example, using login types, error codes, regular expressions, and other parameters.

Thus, tons of messages generated by Windows systems will be checked by Zabbix, and not by our eyes. We can only look at the panel Zabbix Dashboard.
Additionally, I have configured sending notifications to e-mail. This allows you to quickly respond to events, and not to miss events that occurred for example during off-hours.

Monitoring Security Events for Unix Systems

The monitoring system Zabbix also allows you to collect information from the OS log files of the Unix family.

IB events in Unix systems suitable for all

Such security problems of the Unix family of systems are all the same attempts to select passwords for accounts, as well as search for vulnerabilities in authentication tools, such as SSH, FTP and others.

Some critical events in Unix systems

Based on the above, it follows that we need to track actions related to adding, changing and deleting user accounts in the system.
Just an important fact will be tracking login attempts. Changes to key files like sudoers, passwd, etc / rc.conf, the contents of the /usr/local/etc/rc.d directories, the presence of running processes, etc.

Ways to monitor IS in Unix systems

Consider the following example. You need to track logins, failed login attempts, password attempts on the FreeBSD system using the SSH protocol.

All information about this is contained in the log file /var/log/auth.log.
By default, the rights to this file are 600, and it can be viewed only with root privileges. You have to sacrifice a little bit of the local security policy and allow the zabbix user group to read this file:
Change the file permissions:

 chgrp zabbix /var/log/auth.log chmod 640 /var/log/auth.log 

We will need a new data item with the following key:

 log[/var/log/auth.log,sshd,,,skip] 

All lines in the /var/log/auth.log file containing the word ”sshd” will be transferred by the agent to the monitoring server.

You can further customize the trigger with the following expression:

 {Template FreeBSD - SSH:log[/var/log/auth.log,sshd,,,skip].regexp(error:)}|{Template FreeBSD - SSH:log[/var/log/auth.log,sshd,,,skip].regexp(Wrong passwordr:)}&{Template FreeBSD - SSH:log[/var/log/auth.log,sshd,,,skip].nodata(3m)}=0 

This expression is defined as a problem when records selected by the regular expression “error:” appear in the log file. Having opened the history of the received data, we will see errors that occurred during authorization via the SSH protocol.

Here is an example of the last element value for which this trigger is triggered:



Consider another example of security monitoring in FreeBSD OS:
With the help of Zabbix agent, we can check the checksum of the / etc / passwd file.
The key in this case will be the following:

 vfs.file.cksum[/etc/passwd] 

This allows you to control account changes, including changing the password, adding or deleting users. In this case, we will not know what specific operation was performed, but if no one except the server has access to the server, then this is a reason for quick response. If you need a more detailed policy, you can use other keys, such as custom settings.

For example, if we want to get a list of users who are currently logged on to the system, we can use the following user parameter:

 UserParameter=system.users.list, /bin/cat /etc/passwd | grep -v "#" | awk -F\: '{print $$1}' 

And, for example, set the trigger to change in the resulting list.

Or you can use such a simple parameter:

 UserParameter=system.users.online, /usr/bin/users 

So we will see on the Dashboard, who is currently in the system:

Monitoring of information security events on network devices

With Zabbix, you can also very effectively track information security events on Cisco and Juniper network devices using the SNMP protocol. Data transfer from devices is carried out using so-called traps (SNMP Trap).

From an IS point of view, the following events can be identified that need to be monitored — changes in hardware configurations, execution of commands on the switch / router, successful authorization, failed login attempts, and much more.

Monitoring methods

Consider again an example with authorization:
As a stand, I will use the GNS3 emulator with a Cisco 3745 router. I think many people are familiar with this scheme.

First we need to configure sending SNMP traps from the router to the monitoring server. In my case it will look like this:

 login block-for 30 attempts 3 within 60 login on-failure log login on-success log login delay 5 logging history 5 snmp-server enable traps syslog snmp-server enable traps snmp authentication snmp-server host 192.168.1.1 public 

We will send events from Syslog and authentication ladders. I note that successful and unsuccessful authorization attempts are written in Syslog.

Next, you need to configure the reception of the necessary SNMP traps on the monitoring server.
Add the following lines to snmptt.conf:

 EVENT clogMessageGenerated .1.3.6.1.4.1.9.9.41.2.0.1 "Status Events" Normal FORMAT ZBXTRAP $ar $N $* SDESC EDESC 

In our example, we will catch the Syslog traps.

Now you need to configure the data item to collect statistics with the following key:

 snmptrap[“Status”] 

If the trap is not configured on the monitoring server, then the server log will contain approximately the following entries:

 unmatched trap received from [192.168.1.14]:... 

As a result, the received log will reflect information about attempts to log in with detailed information (user, source, localport, and reason in case of failure):



Well, you can configure the trigger to display the event on the Dashboard:

 {192.168.1.14:snmptrap["Status"].regexp(LOGIN_FAILED)}&{192.168.1.14:snmptrap["Status"].nodata(3m)}=0 

In combination with the previous item we will have the following information on the Dashboard:



Similarly to the example described above, you can monitor a large number of events occurring on Cisco routers, to describe which one article is clearly not enough.

I want to note that the above example will not work on Cisco ASA and PIX products, as there is a somewhat different way to work with logging authorization.

Juniper and Syslog

One more example, we will analyze authorization monitoring in JunOS 12.1 for Juniper devices.
Here we will not be able to use SNMP traps, because there is no support for sending traps from Syslog messages. We will need a Unix-based Syslog server, in our case it will be the same monitoring server.

On the router, we need to configure sending Syslog to the storage server:

 system syslog host 192.168.1.1 authorization info 

Now all authorization messages will be sent to the Syslog server, you can of course send all messages (any any), but we don’t need a surplus of information, we send only the necessary ones.

Next, go to the Syslog server
We look tcpdump, whether messages come:

 tcpdump -n -i em0 host 192.168.1.112 and port 514 12:22:27.437735 IP 192.168.1.112.514 > 192.168.1.1.514: SYSLOG auth.info, length: 106 

By default, in the syslog.conf settings, everything that comes from auth.info should be recorded in /var/log/auth.log. Then we do everything in the same way as with monitoring Unix inputs.

Here is an example of a line from the log:



It remains only to configure the trigger on this event as well as it was considered in the example with authorization on the Unix server.

PS

In this way, you can track many events, including such as: saving the device configuration (commit), entering and exiting the configuration edit mode (edit).
I also want to note that in a similar way it can be monitored on Cisco devices, but the way with SNMP traps seems to me faster and more convenient, and the need for an intermediate Syslog server is eliminated.

Conclusion

In conclusion, I would like to note that I will gladly accept comments and additions to this article, as well as interesting suggestions on the use of monitoring information security events using Zabbix.
Thanks for attention. :)