Using custom functions in OSSIM parsers
Good day, dear!
In continuation of my article I want to review and share my experience with the “custom functions” functionality used in OSSIM. These are functions that are intended for processing the information received as a result of parsing (logging) event logs. Processing may consist in the resolution of the name by IP address, the definition of geolocation, and all that is enough imagination. In the example below, I will analyze the use of “custom functions” for additional parsing of the information received.
Suppose that you retrieve event logs from the database (hereinafter - the database), as I described in the article . And it so happened that in one of the database fields you have more than one value, for example, “username” or “IP address”, but the whole message line from which you need to select keywords (for example, the “issued command: ls / root; result: ACCEPT "). And you need to get the command text (ls / root) and the result of its execution (ACCEPT) from this line.
Obviously, the standard functionality available for sources of event logs like “mysql” cannot be done. This is where the custom functions functionality comes to the rescue. With its help from the received line we will be able to select pieces of information that interest us. So let's get started.
Based on the example from the article, it is necessary to select information about the command given (everything that follows “command:”) and the result of its execution (everything that follows after “result:”) from the log line stored in the database in the field “message” . Write the command text in the “userdata3” field, and the result of the command in the “userdata4” field.
An example of a database table:
To solve the problem will be performed:
In order to use your own function in the OSSIM parser, you need to create a new file, for example:
And add functions to the file as:
Functions are written in python.
')
I wrote the following function to extract information about the command given from the event text:
As you can see, this function receives the necessary information using a regular expression (everything that is after "command:" and to the last ";". I say the last one because ";" may also be present in the command body and return it to the OSSIM agent for further processing by the parser.
Similarly, we write the second function and add it to the file:
As a result, the file "/usr/share/alienvault/ossim-agent/plugins/db_logs_func.cfg" looks like:
I will not dwell on the explanation of the entire configuration of the parser, since This is already done earlier in the example .
To notify the parser about the need to use the file with the created functions, add the following line to the [config] section of the parser configuration file:
To use the created functions in the OSSIM parser, a configuration string of the following type is used:
<OSSIM field> = {<function name> (<parameter>)}
With this line we inform the OSSIM agent in which field of the OSSIM event description schema you need to place the information obtained by applying the function to the parameter. And the parameter in our case is the information obtained from the database from the “message” field, i.e. event text “issued command: show arp; result: ACCEPT.
The OSSIM fields in our example will be: userdata3, userdata4
Functions, respectively: "def parse_command" and "def parse_result"
The parameter will be "$ 4"
As a result, the lines that need to be added to the parser's configuration file look like this:
Below is the final fragment (query section) of the OSSIM parser configuration file:
After performing these manipulations, you must restart the OSSIM agent:
After restarting, it will be useful to track the messages in the log file for errors (it has suddenly crept in somewhere):
If everything is done correctly, then in the graphical user interface OSSIM you can see the analysis of the event. Approximately the same as in Figure 1.
Figure 1 - Disassembled events in the OSSIM interface
Here I would like to say that my first thought was to try to pass two parameters to a function in order to record different parts from the source text in the fields userdata3 and userdata4.
For example, passing (text, 1) to receive a command, and (text, 2) - respectively, the result. This decision seems to me the most elegant.
I even wrote a function for this that works out if you run it on the server command line. But the OSSIM agent does not want to take two parameters, only one.
I turned to alienvault with this question, but so far I have not received an answer. If anyone has thoughts on this topic, please write in a personal or comments.
Thank you in advance!
In continuation of my article I want to review and share my experience with the “custom functions” functionality used in OSSIM. These are functions that are intended for processing the information received as a result of parsing (logging) event logs. Processing may consist in the resolution of the name by IP address, the definition of geolocation, and all that is enough imagination. In the example below, I will analyze the use of “custom functions” for additional parsing of the information received.
1. What is it for?
Suppose that you retrieve event logs from the database (hereinafter - the database), as I described in the article . And it so happened that in one of the database fields you have more than one value, for example, “username” or “IP address”, but the whole message line from which you need to select keywords (for example, the “issued command: ls / root; result: ACCEPT "). And you need to get the command text (ls / root) and the result of its execution (ACCEPT) from this line.
Obviously, the standard functionality available for sources of event logs like “mysql” cannot be done. This is where the custom functions functionality comes to the rescue. With its help from the received line we will be able to select pieces of information that interest us. So let's get started.
2. Task statement
Based on the example from the article, it is necessary to select information about the command given (everything that follows “command:”) and the result of its execution (everything that follows after “result:”) from the log line stored in the database in the field “message” . Write the command text in the “userdata3” field, and the result of the command in the “userdata4” field.
An example of a database table:
+---------------------+----------+----------------------+----------+--------------------------------------------+ | date | event_id | event_type | username | message | +---------------------+----------+----------------------+----------+--------------------------------------------+ | 2016-07-22 17:17:05 | 283 | type 1 | net_adm | issued command: show arp; result: ACCEPT | | 2016-07-22 17:17:49 | 284 | suspicious activity | operator | issued command: show arp; result: ACCEPT | | 2016-07-22 17:17:50 | 285 | suspicious activity | admin | issued command: show arp; result: ACCEPT | | 2016-07-22 17:17:51 | 286 | suspicious activity | guest | issued command: show arp; result: ACCEPT | | 2016-07-22 17:17:52 | 287 | type 1 | unknown | issued command: show arp; result: ACCEPT | | 2016-07-22 17:17:53 | 288 | type 1 | valeriy | issued command: show arp; result: ACCEPT | | 2016-07-22 17:17:54 | 289 | suspicious activity | alex | issued command: show arp; result: ACCEPT | | 2016-07-22 17:17:55 | 290 | type 1 | cisco | issued command: show arp; result: ACCEPT | | 2016-07-22 17:17:57 | 291 | suspicious activity | net_adm | issued command: show arp; result: ACCEPT | +---------------------+----------+----------------------+----------+--------------------------------------------+ 3. Decision
To solve the problem will be performed:
- creating a function to highlight information about the team;
- creating a function to highlight information about the result of the command;
- additional setting of the parser (previously created in the example ).
In order to use your own function in the OSSIM parser, you need to create a new file, for example:
/usr/share/alienvault/ossim-agent/plugins/db_logs_func.cfg
And add functions to the file as:
Start Function < > < > End Function
Functions are written in python.
')
3.1. Creating a function to highlight information about the team
I wrote the following function to extract information about the command given from the event text:
def parse_command(input): res = re.search(r'command:.*;', input) return (res.group(0).split(": ")[1].strip(";"))
As you can see, this function receives the necessary information using a regular expression (everything that is after "command:" and to the last ";". I say the last one because ";" may also be present in the command body and return it to the OSSIM agent for further processing by the parser.
3.2. Creating a function to highlight information about the result of the command
Similarly, we write the second function and add it to the file:
def parse_result(input): res = re.search(r'result:\s+\S+', input) return (res.group(0).split(": ")[1])
As a result, the file "/usr/share/alienvault/ossim-agent/plugins/db_logs_func.cfg" looks like:
Start Function parse_command def parse_command(input): res = re.search(r'command:.*;', input) return (res.group(0).split(": ")[1].strip(";")) End Function Start Function parse_result def parse_result(input): res = re.search(r'result:\s+\S+', input) return (res.group(0).split(": ")[1]) End Function
3.3. Additional parser setting
I will not dwell on the explanation of the entire configuration of the parser, since This is already done earlier in the example .
To notify the parser about the need to use the file with the created functions, add the following line to the [config] section of the parser configuration file:
custom_functions_file=/etc/ossim/agent/plugin/db_logs_func.cfg
To use the created functions in the OSSIM parser, a configuration string of the following type is used:
<OSSIM field> = {<function name> (<parameter>)}
With this line we inform the OSSIM agent in which field of the OSSIM event description schema you need to place the information obtained by applying the function to the parameter. And the parameter in our case is the information obtained from the database from the “message” field, i.e. event text “issued command: show arp; result: ACCEPT.
The OSSIM fields in our example will be: userdata3, userdata4
Functions, respectively: "def parse_command" and "def parse_result"
The parameter will be "$ 4"
As a result, the lines that need to be added to the parser's configuration file look like this:
userdata3={parse_command($4)} userdata4={parse_result($4)}
Below is the final fragment (query section) of the OSSIM parser configuration file:
[query] query="select event_id, date, event_type, username, message from data_table where event_id > $1;" #order by event_id desc limit 1 regexp= ref=0 date={normalize_date($1)} plugin_sid={translate($2)} username={$3} userdata1={$4} userdata2={$2} userdata3={parse_command($4)} userdata4={parse_result($4)}
After performing these manipulations, you must restart the OSSIM agent:
/etc/init.d/ossim-agent restart
After restarting, it will be useful to track the messages in the log file for errors (it has suddenly crept in somewhere):
tail -f /var/log/alienvault/agent/agent.log|grep ERROR
If everything is done correctly, then in the graphical user interface OSSIM you can see the analysis of the event. Approximately the same as in Figure 1.
Figure 1 - Disassembled events in the OSSIM interface
4. Improvement
Here I would like to say that my first thought was to try to pass two parameters to a function in order to record different parts from the source text in the fields userdata3 and userdata4.
For example, passing (text, 1) to receive a command, and (text, 2) - respectively, the result. This decision seems to me the most elegant.
I even wrote a function for this that works out if you run it on the server command line. But the OSSIM agent does not want to take two parameters, only one.
I turned to alienvault with this question, but so far I have not received an answer. If anyone has thoughts on this topic, please write in a personal or comments.
Thank you in advance!
Source: https://habr.com/ru/post/306258/
All Articles