MongoDB as a logger for OSSIM
Good day, dear readers! This article was conceived by me as an addition to the wonderful article by Evgeny Sokolov about using MongoDB as a logger for OSSIM (that is, a long-term repository of event logs). I supplemented the original article by Eugene with comments and explanations where, in my opinion, it was necessary for a better understanding.
I allowed myself to copy some parts of the original article by Eugene with the permission of the author himself.
Those who want to use the SIEM system Alienvault OSSIM, or are already using it, probably noticed one of its main differences from the paid counterpart (Alienvault USM) - the absence of the “Logger” component. Thus, the only option for storing the collected event logs is the MySQL database, which is based on OSSIM. But the larger its volume becomes, the slower the search and the entire system begins to work. The best option is to store a small “operational” volume of journals in the OSSIM database. For long-term storage, it's better to use something else, for example, MongoDB.
')
Task: configure OSSIM to send collected event logs for storage in an external database, which is MongoDB.
Initial data:
On the installation process, MongoDB will not stop, because It is well described on the official site of the system .
The scheme of interaction of components is shown in the figure below.
Figure 1 - Diagram of the interaction of components OSSIM and mongoDB
OSSIM consists of two main components: agent (ossim-agent) and server (ossim-server). The agent is engaged in the processing of event logs from different sources and their normalization, i.e. leading to a single format. After the agent normalizes, events are transmitted to the ossim-sever component for processing (correlation, building reports, etc.). After processing the events are placed in the MySQL database. As part of solving the task, the agent will be configured to send normalized events to the mongoDB server, as shown in Figure 1.
A little going into the agent code (written in python), you can see that the agent creates an object of class Event, which gives to the server. The Event object in one of its attributes contains the raw log line from the source. The event is transmitted to the server by the OutputPlugins class, whose code lies in /usr/share/alienvault/ossim-agent/Output.py. If we read this code, we will see that the data output is provided not only to the server, but also to a text file, to a file in csv format, to a SQL database (to any one that the python adoDB module supports). These options can be simply enabled in the agent configuration file in /etc/ossim/agent/config.cfg. Various "conclusions" can be included simultaneously. In this way, parallel transmission of events to an external logger can be enabled.
Here it is necessary to clarify that in order to implement the required functionality, it will be necessary to make edits to the code of the ossim itself, or rather, the three files responsible for processing the event logs by the agent: Output.py, Agent.py, Event.py.
To solve the problem will be performed:
The Output.py, Agent.py, Event.py files are located on the OSSIM server in the / usr / share / alienvault / ossim-agent folder.
Modified to work with mongoDB samples of the files Output.py, Agent.py, Event.py, which can be copied to your OSSIM installation, replacing the original ones (just in case, making a backup copy of them) can be taken via the link in the OSSIM / usr / folder share / alienvault / ossim-agent / »branches 2.1.1. Below will be described the changes made to the original files Output.py, Agent.py, Event.py in OSSIM.
Modify Output.py
First of all, you need to add class code to the Output.py to create an output object. In the GLOBAL IMPORTS section:
Added the last two lines. “From pymongo import MongoClient” - connects the mongoDB client module (obviously, to work with mongo). "From bson import BSON" - connects the BSON codec (binary JSON).
Next, the “OutputESGuard” class is added, which performs the transmission of events to the mongoDB server. It looks like this:
Also in Output.py, the add_esguard_output method was added to the Output class code:
Modification of Event.py
The method to_bson_esguard () (which is used by the event method in output.py) has been added to the Event class code (/usr/share/alienvault/ossim-agent/Event.py). This method (to_bson_esguard ()) is a copy of the existing to_bson method with one difference. In the standard method, the attribute log (it contains the event in its raw form) is declared as binary and is recorded in Base64. We also need text in utf8 there. That is what was changed.
However, not all plugins write payload text. There is, for example, Suricata, which can put binary data there. This does not always happen. Usually, too, the text, but sometimes - a stream of bytes. When trying to push a binary to the server in a field described as text, it returns a parser error utf8. To prevent such an error, pass to the try block. If an error occurs, the event is recoded using the standard to_bson () method and sent to the server again.
Modification Agent.py
In Agent.py, you need to add a section for reading and processing the configuration file (/etc/ossim/agent/config.cfg), checking for the presence of the section [output-esguard] and reading the parameters specified in it. In this section, the IP address and credentials of the mongoDB server will be specified, to which the event logs are sent.
The minimum mongoDB setting implies:
To create a database user, you need to connect to the database from the console of the server on which it is installed:
Creating a user is done with the command:
With the above command, we created a user with the name “ossim”, password “1q2w3e4r”, which has read and write access to the “test” database. According to the ideology of mongoDB, manually creating a database (as is customary in relational databases, the same MySQL) is not necessary. It is created automatically, at the moment when the first record enters it.
Also, for convenience, you can create a user with the privileges of creating and managing accounts of other users (so that we can manage users after we enable authentication and authorization when accessing the database):
The mongoDB installation settings “out of the box” do not allow connecting to the database via the network; therefore, you need to comment out the “bindIp: 127.0.0.1” line in the “net” section of the configuration file "/etc/mongod.conf" (or enter which database will respond):
In addition, these same settings allow anonymous users (O_O) to connect to the database. To enable authentication and authorization mechanisms, you need to add the "authorization: enabled" lines to the "security" section (in mongoDB version 2.x, this is done differently. If you are using version 2.x, see the official documentation on the site):
And restart the server:
Check that the KS user can work with a command from the console:
So, at the moment, we have the ossim-agent component of the ossim server with the configured additional output of events in mongoDB and the database itself, ready to receive data over the network. It remains only to inform the ossim agent where to send the event logs. For this, in the configuration file of the agent (/etc/ossim/agent/config.cfg) you need to add the appropriate section. An example of its contents is shown below:
The configuration speaks for itself: the enable switch, the server's address is the host, the port is the port (default is 27017), the database name is base, the user name is user, and the password is pass.
After editing this configuration file, it is necessary to restart the agent with the command:
If done correctly, the events have already begun to be transmitted to mongoDB. You can see if they appeared there using graphic clients, of which there are many (Google help) or by connecting to mongoDB via the console and running the “show collections” command:
Thanks for attention!
I allowed myself to copy some parts of the original article by Eugene with the permission of the author himself.
1. What is it for?
Those who want to use the SIEM system Alienvault OSSIM, or are already using it, probably noticed one of its main differences from the paid counterpart (Alienvault USM) - the absence of the “Logger” component. Thus, the only option for storing the collected event logs is the MySQL database, which is based on OSSIM. But the larger its volume becomes, the slower the search and the entire system begins to work. The best option is to store a small “operational” volume of journals in the OSSIM database. For long-term storage, it's better to use something else, for example, MongoDB.
')
2. Task statement and source data
Task: configure OSSIM to send collected event logs for storage in an external database, which is MongoDB.
Initial data:
- server with installed OSSIM (version 5.2.2 used, IP: 192.168.0.111);
- server with MongoDB installed (server running Debian 8.5, MongoDB version 3.2 database, IP: 192.168.0.77).
On the installation process, MongoDB will not stop, because It is well described on the official site of the system .
The scheme of interaction of components is shown in the figure below.
Figure 1 - Diagram of the interaction of components OSSIM and mongoDB
OSSIM consists of two main components: agent (ossim-agent) and server (ossim-server). The agent is engaged in the processing of event logs from different sources and their normalization, i.e. leading to a single format. After the agent normalizes, events are transmitted to the ossim-sever component for processing (correlation, building reports, etc.). After processing the events are placed in the MySQL database. As part of solving the task, the agent will be configured to send normalized events to the mongoDB server, as shown in Figure 1.
A little going into the agent code (written in python), you can see that the agent creates an object of class Event, which gives to the server. The Event object in one of its attributes contains the raw log line from the source. The event is transmitted to the server by the OutputPlugins class, whose code lies in /usr/share/alienvault/ossim-agent/Output.py. If we read this code, we will see that the data output is provided not only to the server, but also to a text file, to a file in csv format, to a SQL database (to any one that the python adoDB module supports). These options can be simply enabled in the agent configuration file in /etc/ossim/agent/config.cfg. Various "conclusions" can be included simultaneously. In this way, parallel transmission of events to an external logger can be enabled.
3. Decision
Here it is necessary to clarify that in order to implement the required functionality, it will be necessary to make edits to the code of the ossim itself, or rather, the three files responsible for processing the event logs by the agent: Output.py, Agent.py, Event.py.
To solve the problem will be performed:
- adding code to ossim files (Output.py, Agent.py, Event.py);
- configure mongoDB to receive event logs;
- Enable sending event logs to mongoDB server.
3.1. Adding code to ossim files (Output.py, Agent.py, Event.py)
The Output.py, Agent.py, Event.py files are located on the OSSIM server in the / usr / share / alienvault / ossim-agent folder.
Modified to work with mongoDB samples of the files Output.py, Agent.py, Event.py, which can be copied to your OSSIM installation, replacing the original ones (just in case, making a backup copy of them) can be taken via the link in the OSSIM / usr / folder share / alienvault / ossim-agent / »branches 2.1.1. Below will be described the changes made to the original files Output.py, Agent.py, Event.py in OSSIM.
Modify Output.py
First of all, you need to add class code to the Output.py to create an output object. In the GLOBAL IMPORTS section:
# GLOBAL IMPORTS # import os import re import string import sys import uuid from pymongo import MongoClient from bson import BSON Added the last two lines. “From pymongo import MongoClient” - connects the mongoDB client module (obviously, to work with mongo). "From bson import BSON" - connects the BSON codec (binary JSON).
Next, the “OutputESGuard” class is added, which performs the transmission of events to the mongoDB server. It looks like this:
class OutputESGuard(OutputPlugins): def __init__(self, conf): logger.info("Added ESGuard output") logger.debug("OutputDB options: %s" % (conf.hitems("output-esguard"))) self.conf = conf self.dbhost = self.conf.get('output-esguard', 'host') self.dbport = self.conf.get('output-esguard', 'port') self.dbschema = self.conf.get('output-esguard', 'base') self.dbuser = self.conf.get('output-esguard', 'user') self.dbpass = self.conf.get('output-esguard', 'pass') mongodbURI = "mongodb://" + self.dbuser + ":" + self.dbpass + "@" + self.dbhost + ":" + self.dbport + "/" + self.dbschema try : self.conn = MongoClient(mongodbURI) self.log_db = self.conn[self.dbschema] self.event_coll = self.log_db['logger'] self.activated = True except Exception, e: logger.error(": Error connecting to Mongodb %s" % (e)) def event(self, e): if e["event_type"] == "event" and self.activated: try : self.event_coll.insert_one(BSON.decode(e.to_bson_esguard())) except : logger.error(": Error insert data to mongodb log_coll. Plugin_id is %s. Retry as binary" % (e['plugin_id'])) self.event_coll.insert_one(BSON.decode(e.to_bson())) def shutdown(self): logger.info("Closing ESGuard output ..") self.conn.close() self.activated = False Also in Output.py, the add_esguard_output method was added to the Output class code:
@staticmethod def add_esguard_output(conf): if Output.esguard_output is False: Output._outputs.append(OutputESGuard(conf)) Output.esguard_output = True Modification of Event.py
The method to_bson_esguard () (which is used by the event method in output.py) has been added to the Event class code (/usr/share/alienvault/ossim-agent/Event.py). This method (to_bson_esguard ()) is a copy of the existing to_bson method with one difference. In the standard method, the attribute log (it contains the event in its raw form) is declared as binary and is recorded in Base64. We also need text in utf8 there. That is what was changed.
However, not all plugins write payload text. There is, for example, Suricata, which can put binary data there. This does not always happen. Usually, too, the text, but sometimes - a stream of bytes. When trying to push a binary to the server in a field described as text, it returns a parser error utf8. To prevent such an error, pass to the try block. If an error occurs, the event is recoded using the standard to_bson () method and sent to the server again.
Modification Agent.py
In Agent.py, you need to add a section for reading and processing the configuration file (/etc/ossim/agent/config.cfg), checking for the presence of the section [output-esguard] and reading the parameters specified in it. In this section, the IP address and credentials of the mongoDB server will be specified, to which the event logs are sent.
def init_output(self): ''' Initialize Outputs ''' printEvents = True if self.conf.has_section("output-properties"): printEvents = self.conf.getboolean("output-properties", "printEvents") Output.print_ouput_events(printEvents) if self.conf.has_section("output-plain"): if self.conf.getboolean("output-plain", "enable"): Output.add_plain_output(self.conf) # output-server is enabled in connect_server() # if the connection becomes availble if self.conf.has_section("output-csv"): if self.conf.getboolean("output-csv", "enable"): Output.add_csv_output(self.conf) if self.conf.has_section("output-db"): if self.conf.getboolean("output-db", "enable"): Output.add_db_output(self.conf) if self.conf.has_section("output-esguard"): if self.conf.getboolean("output-esguard", "enable"): Output.add_esguard_output(self.conf) 3.2. Configure mongoDB to receive event logs
The minimum mongoDB setting implies:
- creating a user to connect to the database;
- Configuring the slider to connect to the database via the network;
- enable authentication;
To create a database user, you need to connect to the database from the console of the server on which it is installed:
# mongo MongoDB shell version: 3.2.8 connecting to: admin Creating a user is done with the command:
db.createUser( { user: "ossim", pwd: "1q2w3e4r", roles: [ {role: "readWrite", db: "test"} ] } ) With the above command, we created a user with the name “ossim”, password “1q2w3e4r”, which has read and write access to the “test” database. According to the ideology of mongoDB, manually creating a database (as is customary in relational databases, the same MySQL) is not necessary. It is created automatically, at the moment when the first record enters it.
Also, for convenience, you can create a user with the privileges of creating and managing accounts of other users (so that we can manage users after we enable authentication and authorization when accessing the database):
db.createUser( { user: "uadm", pwd: "1q2w3e4r", roles: [ "userAdminAnyDatabase" ] } ) The mongoDB installation settings “out of the box” do not allow connecting to the database via the network; therefore, you need to comment out the “bindIp: 127.0.0.1” line in the “net” section of the configuration file "/etc/mongod.conf" (or enter which database will respond):
net: port: 27017 # bindIp: 127.0.0.1 In addition, these same settings allow anonymous users (O_O) to connect to the database. To enable authentication and authorization mechanisms, you need to add the "authorization: enabled" lines to the "security" section (in mongoDB version 2.x, this is done differently. If you are using version 2.x, see the official documentation on the site):
security: authorization: enabled And restart the server:
/etc/init.d/mongod restart Check that the KS user can work with a command from the console:
# mongo localhost:27017/test -u ossim -p MongoDB shell version: 3.2.8 Enter password: connecting to: localhost:27017/test 3.3. Enable sending event logs to mongoDB server
So, at the moment, we have the ossim-agent component of the ossim server with the configured additional output of events in mongoDB and the database itself, ready to receive data over the network. It remains only to inform the ossim agent where to send the event logs. For this, in the configuration file of the agent (/etc/ossim/agent/config.cfg) you need to add the appropriate section. An example of its contents is shown below:
[output-esguard] enable=True host=192.168.0.77 port=27017 base=test user=ossim pass=1q2w3e4r The configuration speaks for itself: the enable switch, the server's address is the host, the port is the port (default is 27017), the database name is base, the user name is user, and the password is pass.
After editing this configuration file, it is necessary to restart the agent with the command:
/etc/init.d/ossim-agent restart 4. Conclusion
If done correctly, the events have already begun to be transmitted to mongoDB. You can see if they appeared there using graphic clients, of which there are many (Google help) or by connecting to mongoDB via the console and running the “show collections” command:
# mongo localhost:27017/test -u ossim -p MongoDB shell version: 3.2.8 Enter password: connecting to: localhost:27017/test > show collections logger.20160810 logger.20160811 logger.20160812 logger.20160813 logger.20160814 logger.20160815 Thanks for attention!
Source: https://habr.com/ru/post/307746/
All Articles