Dirty place provider: the project blondemine

Our company has a distributed sales network. Communication office with the store provides a router on which VPN is configured to the center. And so, from a certain point on, this connection became extremely poor-quality, because of the squall of packets on the 53rd DNS port. Communication, though improved after the introduction of blocking rules of the firewall, but the attacks did not stop.

I asked the provider to solve the problem on his side. To which he received the answer made in the headline: "Well, you understand, the Internet is a dirty place." And then I decided to fight this phenomenon on my own.

The result was a collection and analysis system for unauthorized network packets.
And in order not to bore the reader with technical details, I will tell the most interesting things right away, but I recommend reading those who wish to repeat or improve it.
')
As a result of almost two months of operation, the system received almost 200 million records. Most (98%) are attacks of the type of DNS amplification, which were discussed more than once in Habré [1] , [2] . Moreover, the attacks do not start immediately, but after some time, enough for a new public address to get into the database of scanning Internet bots. In the rest of the events, a large segment of attacks on port 23 is highlighted. As I found out, these are Hikvision's Chinese DVR systems scattered all over the world and scanning the entire Internet for telnet connections. On the third of them, by the way, just the factory login and password. And all the rest is already man-made port interruptions, attempts to log in, interrogate by snmp and so on.

How I came to this

First you had to configure a firewall on all remote routers. Brought the “exemplary” set of rules, checked and started to customize. On the eighth router, I got bored and I looked into the device log: “vk.com cannot connect to the manager’s computer”, “youtube.ru does not connect to the cash register” and so on and so forth. But there were interesting entries: “src = 1.1.1.1: 34567 dst = Zyxel: 23 dropped [4 times]”, “src = 2.2.2.2: 45678 dst = Zyxel: 80 dropped [12 times]”.

I decided to scan these addresses with Nmap-ohm: the 80th and 22nd ports are open. The browser said that the first address is a webcam in a meeting room of some Ukrainian company (judging by geoip and whois), and the second is a certain Indian ubiquiti router. The factory login and password worked only on the webcam, but with the router the factory settings were left “only” on ssh!

Well, what kind of "blonde" do you need to be in order to parade the entire Internet with a device with a factory login and password? And I decided to collect such “blondes” in one place and study. And in the future, implement the following algorithm:

attacking attackers - passive analysis - active analysis - response

Step One: Collect Attackers

Central Journal

In Zyxel routers, you can send a log to the central syslog server. For this, I installed Linux Debian, and on it rsyslog:

sudo apt-get update sudo apt-get install rsyslog 

However, simple installation is not enough. It is necessary to allow syslog to receive messages from outside:



Well, in order for this port to catch data from the Internet, you also need to broadcast on the border router. In my case at Cisco, I did this:

 ip nat inside source static udp {syslog_IP} 514 {public_IP} 514 extendable 

Now you need to configure the remote router to send messages. In my case, this is Zyxel zywall 2Plus or USG20.

Models of these routers differ quite strongly, both outside and inside. However, the whole point of the setup comes down to disabling all logs, except for the firewall, and sending events to the syslog server. Here's what it looks like on the Zyxel USG 20:
firewall configuration:



configure syslog log events:



configure syslog server address and type:



Even in the device settings, you must set a meaningful host name - for ease of sorting.

First results

If everything is left like this, then a text file local5 will be created on the syslog server with all the events corresponding to the last rule of the firewall - that is, attacks on this router:

 Sep 27 17:34:54 2017 GW18PUB src="5.188.203.30:54193" dst="XXXX:8173" msg="invalid state detected, DROP" note="ACCESS BLOCK" user="unknown" devID="b0b2dcc63eec" cat="Firewall" class="Access Control" ob="0" ob_mac="000000000000" dir="ANY:ANY" protoID=6 proto="others" Sep 27 17:35:02 2017 GW36PUB src="212.83.176.116:51855" dst="YYYY:4287" msg="invalid state detected, DROP" note="ACCESS BLOCK" user="unknown" devID="b0b2dcc63eed" cat="Firewall" class="Access Control" ob="0" ob_mac="000000000000" dir="ANY:ANY" protoID=6 proto="others" Sep 27 17:35:17 2017 GW18PUB src="212.83.176.116:51855" dst="XXXX:3875" msg="invalid state detected, DROP" note="ACCESS BLOCK" user="unknown" devID="b0b2dcc63eec" cat="Firewall" class="Access Control" ob="0" ob_mac="000000000000" dir="ANY:ANY" protoID=6 proto="others" 

The file will grow indefinitely until it eats the entire logical partition, so you need to configure the rotation of the log. This can be done using logrotate scripts - split the log into days, daily archive and delete old archives. However, I had a better idea - to take only the necessary from the record, split it into fields and transfer them to the database.

Database and Log Handler

As a DBMS, I installed postgresql - because I did not know anything about working with it at that time, but I really wanted to get to know each other.

 sudo apt-get update sudo apt-get install postgresql-9.5 

After installation, I connected to postgres:

 sudo -u postgres psql 

Created a new user:

 CREATE USER pgmaxa WITH password 'strongpass'; 

And the base:

 CREATE DATABASE blondemine; GRANT ALL ON DATABASE blondemine TO pgmaxa; 

The first “wedro” table should contain the time of the event, the name of the router, the address of the attacker and the port attacked. And everything would be fine, but storing the attacker's address in text form seemed to me wrong. As a result, I installed the IP4R extension - it did not work out on the first attempt, but if I’m interested, I’ll tell you. As a result, the table was created as follows:

 CREATE TABLE wedro ( rt timestamp without time zone, gw character varying(9), ad ip4, pt integer, dtst time without time zone NOT NULL DEFAULT now() ) 

Log handler

Now it’s time to create a local5 handler that will fill my database. It was not written in one day, and even underwent five revisions, so I’ll give you the latest working version:



The essence of this “script” is:

• we process the journal line by line
• we take only lines with the phrases "Access Control" and "default"
• (!) remove the four-digit year and time gaps in the event
• From the fifth field we delete “src =”, and write the address in the variable HA (HackerAddress)
• from the sixth field we delete “dst =”, and we write the port to the variable OP (OurPort)
• check the length of the router name and port number for correctness
• collect the data in the string for the insert query
• cut the file with the request for two characters and complete the “;”
• rename and paste into database

I put the exclamation mark for a separate comment. The fact is that the format of magazines Zywall2Plus and USG20 are different in that one writes a year, and the other for some reason does not. To bring these records to a general form, I delete the year from all the records, and then add the year value from the system. This is actually not very correct, and after a while I realized that I had to do it differently. For each format of the journal it would be necessary to make a certain processing template as a module. However, there was no time and wrote how it goes.

In order for the database to be filled automatically, in the rotation settings of the /etc/logrotate.d/ logs, you need to create a rule for local5, which sets the rotation by file size:

 /var/log/local5 { rotate 1 prerotate /usr/sbin/handmade/logtopostgres.sh endscript postrotate invoke-rc.d rsyslog rotate > /dev/null endscript size 2190000 missingok notifempty compress } 

However, rotation for some reason did not always work, and I did a forced rotation every 5 minutes in the cron scheduler:

 */5 * * * * root /usr/sbin/logrotate /etc/logrotate.d/loc5 

Result

As a result, the database is really filled with an automatic machine - in two days, about four million entries from several routers.

Step Two: Passive Analysis

From the very beginning, it became clear that there were too many records with port 53 to see something else. As a result, nothing better than deleting them, I have not yet invented. A trigger on the AFTER INSERT was hung on the wedro table:

 CREATE OR REPLACE FUNCTION public.deldns() RETURNS trigger AS $BODY$ BEGIN DELETE FROM wedro WHERE pt=53 OR ad::text like '192.168%'; RETURN NEW; END; $BODY$ CREATE TRIGGER sweep AFTER INSERT ON public.wedro FOR EACH STATEMENT EXECUTE PROCEDURE public.deldns(); 

Result

With such a regular cleaning, the base ceased to grow at an alarming rate and there was an opportunity to find something in it. True, there is also a need for AUTOVACUUM to free up once used resources.

Unique addresses

Attacks from the same addresses are repeated quite often. Therefore, I decided that they should be collected in a separate table. Since the data is updated every five minutes, there is no point in doing this in the form of a trigger - as a result, we had to master the installation and operation of the pgagent scheduler.

Two more ipid and wedro_old tables were created for this purpose:

 CREATE TABLE ipid ( id bigint NOT NULL DEFAULT nextval('ipid_id_seq'::regclass), ad ip4 NOT NULL, dtst timestamp without time zone NOT NULL DEFAULT now() ); CREATE TABLE wedro_old ( rt timestamp without time zone, gw character varying(9), ad ip4, pt integer, dtst time without time zone NOT NULL DEFAULT now() ); 

And in pgagent-e, an hourly task was set up to fill these tables:

 INSERT INTO ipid (ad) SELECT DISTINCT ad FROM wedro WHERE dtst > (select max(dtst) from ipid)::time AND ad NOT IN (select ad from ipid); INSERT INTO wedro_old SELECT * FROM wedro; TRUNCATE table wedro; 

Step Three: Active Analysis

Hikvision Cameras and DVRs

The remaining entries turned out to be quite a lot of attacks on the 23rd telnet port. Moreover, a number of addresses did this many times and did not scan any other ports. If you type this address in the browser, the form for entering credentials will open:



in some cases, factory admin / 12345 is allowed into the interface of the digital tape recorder Hikvision. It is also interesting that on some devices there is an undocumented open port 9527. Here is an example for the active analysis phase:

We are looking for addresses attacking only port 23:

 SELECT ad INTO telnetz FROM wedro_old WHERE ad IN ( SELECT a.ad FROM (SELECT ad,pt from wedro_old group by ad,pt) AS a GROUP BY a.ad HAVING count(*)=1 ) AND pt=23 GROUP BY ad,pt; 

We use the resulting list in Nmap-e to search for vulnerable DVRs:

 #!/bin/bash nmap \ -n -Pn -p9527 --open `\ echo "select * from telnetz limit $1 offset $2;" | \ psql -U pgmaxa -d blondemine -t -h syslog_IP` | \ grep for | sed 's/^.*for //g' 

The limit and offset expressions are used here because of the scanner’s limitations on the number of targets. On an unlimited list, it fails.

Step Four: Action

For the nodes found in the previous step, I also wrote a expect script to check the factory credentials and disable the 23rd port scanner. However, I was far enough away from my original goal of actively preventing unauthorized access to my routers. And I think it is time to summarize.

Conclusion

The Internet is a really “dirty” place and my system allows it to be quantified. I think, from what I brought here, it is clear that the system is far from perfect and needs to be improved: you need an interface, device templates, the integration of all modules into a single installation package, and much more. But the system is already running, assembled from available and free components. I do not exclude that systems like mine exist in commercial execution. However, I doubt that all modules of such systems will be available for customization and modification.