Add Splunk Free IP Reputation Feeds
In the Splunk application database, there are many solutions that allow you to make an Enrichment and add information that a particular IP address looks suspicious and "lit up" in a particular reputation base. However, these applications are either paid (for example, Recorded Future App , Kaspersky Threat Feed Feed App ), or very slow ( IP Reputation App , at the time of this writing is also not fully available due to Maintenance since February), so we decided to develop your own open source RST Cloud Threat Database Add-on for Splunk plugin, which will allow you to collect disparate information from open sources into a single database and provide answers to streams of tens of thousands of requests per second.
To create the plugin, a small set of Python scripts was developed that enriches the data when searching in Splunk on the fly, requesting the necessary information in an external database. As a database, we used the Key-Value storage of Redis, which holds all the values in the RAM and practically does not depend on the read / write speed from the disk subsystem. The plugin is published on github and is open to suggestions for improvement.
Our performance tests on a virtual machine with 2 Intel® Xeon® E5-2630 cores and 4 GB of RAM showed that, taking into account all the overhead from Python 2.7, virtualization on conventional hardware and Splunk itself, the throughput at 300K entries in Redis averages 25K RPS, which is enough for many tasks. It should immediately draw your attention to the fact that these figures are obtained when using Redis "out of the box" without additional optimization and clustering. Also, the search script does not yet use pipeline mechanisms when working with the database.
')
For example, using a plug-in allows you to define a web form spammer or connect to a site from infected IP addresses.
You can quickly remove all "dangerous" clients from the console:
By displaying the data, we show their source, the categories in which the given IP appeared in different databases and the cumulative assessment of Threat Score.
You can use the macro, which is more convenient to work with:
To simplify, Redis can stand directly on Splunk Head, or it can be moved to another server or server cluster. In addition, the RST Cloud Threat Database Add-on includes several scripts that automatically download reputation databases from various sources and import them into Redis.
Today there are a lot of reliable and open sources for collecting suspicious and dangerous IP addresses. For example, a plugin allows you to work with more than 15, including:
The base can include both individual addresses that are searched by IP key: red.smembers ('ip:' + clientip) , and subnets processed by the script in the for i in red.sscan_iter loop (name = 'net: index', match = str (ip.words [0]) + '*', count = 500) .
We now turn to the installation. Description in steps:
We will omit the first step, firstly, there are a lot of manuals, and, secondly, it is put on debian by one command apt-get install -y redis-server with basic settings.
To solve problems with dependencies on Python libraries, just run:
It is worth noting that Splunk uses its own Python inside itself, which should not be modified, so it’s better to install all the extras in the Python that lives in the operating system.
Depending on how you installed Redis, you may need to fix the addresses and connection ports in the scripts in the $ SPLUNK_HOME / etc / apps / threatDB / bin directory.
Splunk main search script: redisworker.py
DB cleanup script: threat_flushdb.py
Download script updated IoC: threatuploader.py
IoC Script Loader from various sources: start_threatupload.sh
Next, choose a temporary directory:
And we set up a CRON scheduler, for example, through the / etc / crontab file:
In a production environment, instead of root, it is better to use another account. It is also worth noting that the default update once a day is sufficient, since the TTL for records in the database is 48 hours.
Paid solutions provide a whole set of indicators of compromise, including IP addresses, domain addresses, hashes and file paths, mutex names, etc., which allows identifying malware activity in an organization, while in RST Cloud we concentrate on the web, so for now our The plugin allows you to work only with IP addresses. In the future, we plan to refine the plugin in terms of performance and expand its functionality, including adding feeds of reputation databases with DNS names, thereby expanding its scope.
To create the plugin, a small set of Python scripts was developed that enriches the data when searching in Splunk on the fly, requesting the necessary information in an external database. As a database, we used the Key-Value storage of Redis, which holds all the values in the RAM and practically does not depend on the read / write speed from the disk subsystem. The plugin is published on github and is open to suggestions for improvement.
Our performance tests on a virtual machine with 2 Intel® Xeon® E5-2630 cores and 4 GB of RAM showed that, taking into account all the overhead from Python 2.7, virtualization on conventional hardware and Splunk itself, the throughput at 300K entries in Redis averages 25K RPS, which is enough for many tasks. It should immediately draw your attention to the fact that these figures are obtained when using Redis "out of the box" without additional optimization and clustering. Also, the search script does not yet use pipeline mechanisms when working with the database.
')
For example, using a plug-in allows you to define a web form spammer or connect to a site from infected IP addresses.
You can quickly remove all "dangerous" clients from the console:
sourcetype = Web: *: access_log host = www.demo.demo | fields clientip | dedup clientip | lookup local = true lookupthreat clientip OUTPUT threatscore threatsource threatcategory | where threatscore> 0
By displaying the data, we show their source, the categories in which the given IP appeared in different databases and the cumulative assessment of Threat Score.
You can use the macro, which is more convenient to work with:
| `threatDB (clientip)`
To simplify, Redis can stand directly on Splunk Head, or it can be moved to another server or server cluster. In addition, the RST Cloud Threat Database Add-on includes several scripts that automatically download reputation databases from various sources and import them into Redis.
Today there are a lot of reliable and open sources for collecting suspicious and dangerous IP addresses. For example, a plugin allows you to work with more than 15, including:
- Sblam! - base of web spammers in blogs, forums and comments
- StopForumSpam - the base of web spammers in blogs, forums, wiki
- CINS Score - Sentinel IPS base, which the company shared with the community
- Blocklist.de - base with addresses attacking Postfix, SSH, Apache, Spambots, irc-Bots, Reg-Bots, DDos and others
- Ransomware Tracker - CnC server addresses for Ransomware (ransomware programs)
- AlienVault OTX - open feeds from the popular SIEM
- Binary Defense - the specialized provider Theat Intelligence
- EmergingThreats - we have integrated ProofPoint feeds to block firewalls
- Arbor ATLAS - addresses from the most famous company from which DDoS attacks are coming
- Botvrij - malware compromised addresses
- Tor Project - addresses of Tor networks.
The base can include both individual addresses that are searched by IP key: red.smembers ('ip:' + clientip) , and subnets processed by the script in the for i in red.sscan_iter loop (name = 'net: index', match = str (ip.words [0]) + '*', count = 500) .
We now turn to the installation. Description in steps:
- Installing Redis
- Install the required libraries
- Correction of lines for connection in scripts
- Configuring a CRON task to update the database IP Reputation
We will omit the first step, firstly, there are a lot of manuals, and, secondly, it is put on debian by one command apt-get install -y redis-server with basic settings.
To solve problems with dependencies on Python libraries, just run:
$ wget bootstrap.pypa.io/get-pip.py
$ python get-pip.py
$ sudo pip install redis
$ sudo pip install netaddr
It is worth noting that Splunk uses its own Python inside itself, which should not be modified, so it’s better to install all the extras in the Python that lives in the operating system.
Depending on how you installed Redis, you may need to fix the addresses and connection ports in the scripts in the $ SPLUNK_HOME / etc / apps / threatDB / bin directory.
Splunk main search script: redisworker.py
sys.path.append ("/ usr / local / lib / python2.7 / dist-packages") # Path to redis-py module
redis_server = '127.0.0.1'
redis_port = 6379
DB cleanup script: threat_flushdb.py
redis_server = '127.0.0.1'
redis_port = 6379
Download script updated IoC: threatuploader.py
redis_server = '127.0.0.1'
redis_port = 6379
IoC Script Loader from various sources: start_threatupload.sh
base_dir = / opt / splunk / bin / scripts / threatDB
python_bindir = / usr / bin
Next, choose a temporary directory:
$ mkdir -p / tmp / threatsupload
And we set up a CRON scheduler, for example, through the / etc / crontab file:
2 0 * * * root $ SPLUNK_HOME / etc / apps / threatDB / bin / start_threatupload.sh / tmp / threatsupload
In a production environment, instead of root, it is better to use another account. It is also worth noting that the default update once a day is sufficient, since the TTL for records in the database is 48 hours.
Paid solutions provide a whole set of indicators of compromise, including IP addresses, domain addresses, hashes and file paths, mutex names, etc., which allows identifying malware activity in an organization, while in RST Cloud we concentrate on the web, so for now our The plugin allows you to work only with IP addresses. In the future, we plan to refine the plugin in terms of performance and expand its functionality, including adding feeds of reputation databases with DNS names, thereby expanding its scope.
Source: https://habr.com/ru/post/324340/
All Articles