Geekly Articles each Day
📜 ⬆️ ⬇️

Ossim. Creating a simple plugin for parsing logs

This material will be useful to novice administrators who are interested in the OSSIM system.
The installation of the system has already been described in detail in the Habré , so at this moment I will not stop.
The article describes the procedure for writing a plug-in for processing event logs received by the OSSIM system using the syslog protocol from an application running on a remote machine. It should be noted here that this application is not supported out of the box by OSSIM. Thus, this instruction is suitable for developing a plug-in for any of your applications that use the syslog protocol to send event logs.
As a basis, the vendor’s documents were used:
Alienvault. Building collector plugins
Alienvault. How to create a data source plugin


1. Baseline


1.1 Architecture

Server with installed OSSIM (hereinafter - OSSIM Server), having the IP address 192.168.0.111 and the name alienvault.
Server running CentOS 6.6 (hereinafter referred to as Server) with IP address 192.168.0.54 and cnc-2 name. An application is installed on the Server (let's call it bw_lists by convention), which sends event logs in real time via the syslog protocol to the OSSIM Server.
The application is a modification of the standard shell that filters user-entered commands.



Figure 1 - Solution Architecture


1.2 Description of the structure of the event

Event format:
<syslog date> <host name> bw_lists [<session number>]: user: <user name> ip: <Server IP> ip_tko: <target system IP> IN: <user command executed> OUT: Res: <report execution or failure> <command text> <target system output>
In each event sent by syslog the application includes:
1) user session number (in square brackets immediately after the word bw_lists)
2) the name of the user who executed the command
3) IP addresses of the Server and the target system on which this command should work
4) a command executed by the user (for example, "configure terminal")
5) report on the performance or failure to execute the command given by the user (ACCEPT or DENY)
6) console output (for example, "Enter configuration commands, one per line. End with CNTL / Z")
An example of events received from the application:


Nov 20 14:15:33 cnc-2 bw_lists[19025]: user:oper1 ip:192.168.0.54 ip_tko:192.168.1.104 IN: configure terminal OUT: Res: ACCEPT configure terminal Enter configuration commands, one per line. End with CNTL/Z. Nov 20 14:15:39 cnc-2 bw_lists[19025]: user:oper1 ip:192.168.0.54 ip_tko:192.168.1.104 IN: interface Gi0/1 OUT: Res: ACCEPT interface Gi0/1 ^ % Invalid input detected at '^' marker. Nov 20 14:16:29 cnc-2 bw_lists[19025]: user:oper1 ip:192.168.0.54 ip_tko:192.168.1.104 IN: exit OUT: Res: ACCEPT exit

2. Tasks


1) Configure reception of event logs via the syslog protocol from the Server on the OSSIM Server and write them to a file;
2) Configure parsing (parsing) of the received event logs in the OSSIM system. It is necessary to extract the following information from them:



3. Decision


3.1 Setting up the OSSIM Server to receive event logs from the Server

As part of this task, rsyslog will be configured on the OSSIM server to receive event logs that come from the server using the syslog protocol.
To do this, edit the file /etc/rsyslog.conf and add the line to it:


 if $programname contains 'bw_lists' then -/var/log/SR/bw-list-log.log

And restart rsyslog for the changes to take effect:


 /etc/init.d/rsyslog restart

Thus, events received by the OSSIM Server from the Server and containing "bw_lists" will be recorded in the /var/log/SR/bw-list-log.log file. Next, we configure the parsing of events from this file by OSSIM.


3.2 Configuring parsing (parsing) and displaying the received events in OSSIM

The process of setting up the analysis of events in OSSIM is divided into two stages:



The first file is responsible for parsing the log itself and distributing the information received by the fields of the event description scheme used by OSSIM, the second file is used to display the types and classes of events in the OSSIM interface, as well as to prioritize events.
To describe events, OSSIM uses a scheme consisting of several dozen fields, such as: date, sourceIP, destinationIP, username, userdata1-9, etc. Details of the scheme given in off. OSSIM documentation.


3.2.1 Creating a configuration file-parser for event logs

Parser files are located on the OSSIM server in the / etc / ossim / agent / plugins / directory
Create a new file with the name bwlistlog.cfg (the name can be chosen arbitrarily, it doesn’t affect anything and doesn’t appear anywhere else. However, the name must match the <alphanumeric combination without spaces> .cfg).
In each file several sections should be allocated (the section names are enclosed in square brackets), namely:


[DEFAULT]
In this section, the plugin number is present. According to the recommendation of the vendor, the plugin number should be selected from the range of 9000 - 10000. This number should be unique for this OSSIM installation.
You can find out if the number you chose for the plugin is as follows:
grep "plugin_id = <plugin number>" / etc / ossim / agent / plugins / *
for example


 grep "plugin_id=9002" /etc/ossim/agent/plugins/*

[config]
This section indicates:



[translation]
In this section, you can set the "translation" of the values ​​of certain event fields into numbers. Further, these numbers can be used to specify classes of events, for clarity in the graphical interface. For details, see section 3.2.2.


The last section, called in the example [bwlistlog], is the first message parsing rule. The names of sections - the rules can be arbitrary format. It is recommended to set it as follows: [<number> - <source name>], for example [1 - applog], [2 - applog], etc.
The fact is that within the framework of one plugin you can set several rules, i.e. message parsing schemes. This is done in order to handle events that have a different structure (if all the events from the source cannot be parsed using the same parsing rule).
If there are several rules, OSSIM compares the message with each of them in turn, and the queue is arranged in alphabetical order. Example - there are 3 rules with the names [a-rule], [b-rule], [c-rule]. The first to check is compliance with the rule a-rule, then b-rule, etc. Therefore, it is better to call the more general rules in such a way that they are triggered last in the queue.
Contents of the bwlistlog.cfg file with comments:


 [DEFAULT] #    (  OSSIM - plugin.) plugin_id=9002 [config] #  plugin'.     - detector  monitor.    syslog   detector type=detector #   enable=yes #  . log -  .    database (   ), sdee ( cisco), snortlog ( snort), wmi (        Windows   wmi) source=log #    ,     . location=/var/log/SR/bw-list-log.log #   OSSIM      .      create_file=true #       -  OSSIM.  ,     OSSIM,    ,    - process= start=no stop=no startup= shutdown= #      ""   [translation] ACCEPT=1 DENY=2 #   (,   , )  . [bwlistlog] #  .    event_type=event #  ,      regexp="^((?P<date>\S+\s+\d+\s+\d+:\d+:\d+)\s+(?P<sensor>\S+)\s+bw_lists\[(?P<session>\d+)\]\:\s+user\:(?P<user>\S+)\s+ip\:(?P<sr_node>\S+)\s+ip_tko\:(?P<tko_node>\S+)\s+IN\:(?P<message>.*)\s+OUT\:\s+Res\:\s+(?P<result>\S+).*)" #    ,     ,      OSSIM.    3.2.3 device={$sr_node} date={normalize_date($date)} plugin_sid={translate($result)} src_ip={$tko_node} username={$user} userdata1={$message} userdata2={$session}

In this regular expression, we select (using round brackets) the information we want to extract and assign an arbitrary tag to it (using a question mark, the letter P and brackets? P <>), which we can then access to it. Example:


 (?P<date>\S+\s+\d+\s+\d+:\d+:\d+)

means that the information at the beginning of a line that falls under the expression \ S + \ s + \ d + \ s + \ d +: \ d +: \ d + will be addressed by the "date" tag. Further, we inform that in the "date" field of the OSSIM scheme we want to put the value of our "date" field after performing its normalization (OSSIM is able to convert the date in different formats to its single):


 date={normalize_date($date)}

For testing regular expressions you can use, for example, this tool . Keep in mind that this tool does not know how to work with tags. Therefore, having tested your regular expression in it, you will need to add them.
It is also necessary to clarify the "plugin_sid" field. Events from one source - with the same "plugin_id" - may belong to different classes, for example, "ACCEPT" and "DENY" for our example. "plugin_sid" just points to the class identifier, and the line:


 plugin_sid={translate($result)}

tells the parser to go to the [translate] section to perform the translation of the information received by the "result" tag. "ACCEPT" is broadcast in unit, "DENY" - in two. How and why to assign classes of events - in the next section.
Yes, "plugin_sid" can also be assigned manually for each parsing rule, specifying, for example:


 plugin_sid=1

3.2.2 Adding information about the parser and event type (s) to the OSSIM database


After writing the configuration file of the parser, you need to add information about it to the OSSIM database.
This is done by writing a mysql script and running it. Examples of scripts are on the OSSIM Server in the folder:
/ usr / share / doc / ossim-mysql / contrib / plugins /
The name of the script can be arbitrary.
For clarity, just give the script from our example:


 DELETE FROM plugin WHERE id = "9002"; DELETE FROM plugin_sid where plugin_id = "9002"; INSERT IGNORE INTO plugin (id, type, name, description) VALUES (9002, 1, 'bw-lists', 'Bash filtering'); INSERT IGNORE INTO plugin_sid (plugin_id, sid, category_id, class_id, name) VALUES (9002, 1, NULL, NULL, 'Command Accepted'); INSERT IGNORE INTO plugin_sid (plugin_id, sid, category_id, class_id, name) VALUES (9002, 2, NULL, NULL, 'Command Denied');

The first three lines clear all old references to "plugin_id" numbered 9002 from the database and create it again.
The following two just describe the event classes.
As you remember from the previous section, "plugin_sid" is calculated by correlating information obtained from the event using the parser under the "result" tag (maybe "ACCEPT" and "DENY") and then transforming it into the [translate] section (in the configuration file parser). Thus, we have two classes of events 1 - "ACCEPT", 2 - "DENY".
It is about these event classes that the OSSIM database strings are reported:


 INSERT IGNORE INTO plugin_sid (plugin_id, sid, category_id, class_id, name) VALUES (9002, 1, NULL, NULL, 'Command Accepted'); INSERT IGNORE INTO plugin_sid (plugin_id, sid, category_id, class_id, name) VALUES (9002, 2, NULL, NULL, 'Command Denied');

Those familiar with the syntax "mysql" will easily understand them. For the rest, I will explain that events classified as "ACCEPT" in the OSSIM interface will appear with the name "Command Accepted", and "DENY", respectively - "Command Denied". Here you can also set the categorization parameters for events of this type of sources (category_id, class_id).
After creating the script, it is necessary to load it into the OSSIM database with the command:


 # ossim-db < /usr/share/doc/ossim-mysql/contrib/plugins/bw-lists.sql

3.2.3 Enable the plugin

You can enable the created plugin as follows:
1) connect to the OSSIM server via SSH, go to the menu: "Configure sensor" - "Configure Data Source Plugins";
2) find the plugin "bw-lists" in the list and put the value "*" on it, click "OK";
3) return to the root menu with the "Back" button;
4) select the "Apply all Changes" option.
New plugin ready to handle event logs.
However, if the events arrive, and in the OSSIM interface, specifying sorting by plugin type, you do not see them, then ....


4. Troubleshooting


1) check that the log file specified in the parser configuration file exists and records are updated in it - i.e. logs coming in;
2) check that the event format fits the regular expression you have written;
3) check the file /var/log/alienvault/agent/agent.log for errors related to your plugin:


 grep ERR /var/log/alienvault/agent/agent.log|grep 9002 grep Discard /var/log/alienvault/agent/agent.log|grep 9002 grep Warn /var/log/alienvault/agent/agent.log|grep 9002

')

Source: https://habr.com/ru/post/271319/

More articles:


All Articles