Protection against DDOS attacks with improvised means. Getting access to your server

Recently, our site is often subjected to quite powerful DDOS attacks, by the way the last attack was the largest one lately, according to our estimates, the size of the botnet is about 10 thousand machines, the power is 100 Mbits / s.

The attack was noticed even by Kaspersky Lab, and offered its assistance in repelling, for which they were grateful. True, by that time we had independently found a solution that blocks the attack. Actually about this decision and will be discussed.

It all started last Friday at five o'clock in the evening, and lasted until Monday lunchtime. The weekend passed, for an exciting activity on shooting bots. I had to sweat a bit until a working solution was found to counter the attack.
')
The attack was an HTTP Flood type. The system on which we operate the site is Apache for Linux. We have written several scripts that will be given in the text of the article. In principle, a similar approach can be applied to Windows / IIS.

I will try to tell you what main steps we took to repel the attack, and what problems arose along the way:

Getting access to your server

Due to the high load caused by the attack, connection to the server becomes impossible. The output is a reboot, plus a good reaction to try to connect to the restarted machine, and disable the service that is being attacked , and analyze the attack. But with a powerful attack, even after a reboot, the connection is very problematic. Sometimes it was necessary to restart the server several times while it was possible to log in to the system.

After it turned out to log into the system, and turn off the Apache (service httpd stop), you need to remove the Apache launch at system startup. This will give you the opportunity to access the machine by rebooting if something goes wrong. This is done using the command:

# chkconfig httpd off

The solution is not perfect, but it will go first.

Automatic blocking of the attacked service at high system load

Rebooting the server, with each new wave of attack, is a pretty bad decision, because it's all the time that plays against us.

After some thought, a way was found. If the load increases above a certain critical level, block the attacked service with a firewall (in our case, the 80th port).

Actually, a universal script was written that does what was intended. The script call is as follows:

# blockOnHighLoad.sh turnOn80Port 5 turnOff80Port

Two commands are passed to the script, and the maximum system load level at which we need to do something. In this case, when load average reaches 5, a command is launched that blocks the 80th port with a firewall. When the load returns to the normal level, the port opens again.

As a matter of fact, for simple cases, you can describe actions directly when called, for example, the previous command without using external scripts:

blockOnHighLoad.sh "/sbin/iptables -D INPUT -p tcp -m tcp --dport 80 -j DROP" 5 "/sbin/iptables -A INPUT -p tcp -m tcp --dport 80 -j DROP"

Those. we directly block / unblock the port using iptables.

In more complex cases, for example, when an attack goes on several services at once, or you need to perform several steps at once, you can use external commands.

So along with blocking a single port, it is useful to completely “close” the machine, leaving only the port for connection via SSH. Such an action is advisable to carry out with the "care" loading into "space". Here is the second useful command:

blockOnHighLoad.sh "" 50 "blockAllExcept.sh 22"

Here, after reaching the load average value of 50 units, we close all that is possible, with the exception of SSH (port 22). The opening of the port is made in this case in manual mode.

Now it remains to insert all this stuff into autoload of the system. Plus you need to run httpd disabled. To do this, we added the following commands to the /etc/rc.local initialization script:

 blockOnHighLoad.sh "/sbin/iptables -D INPUT -p tcp -m tcp --dport 80 -j DROP" 5 "/sbin/iptables -A INPUT -p tcp -m tcp --dport 80 -j DROP" >> /var/log/blockOnHighLoad-5-80.logs 2>&1 & blockOnHighLoad.sh "" 50 "blockAllExcept.sh 22" >> /dev/null & service httpd start 

Why not leave the normal httpd launch via the system services start? Because it will start in front of the autoblocker, and with a “good” attack, immediately put the system in and the initialization script may fail.

I note that the script prints a small diagnostic log. Messages are displayed when the “mode” of work is switched and the current load and mode are printed. AT
In the example above, this log is saved to the /var/log/blockOnHighLoad-5-80.logs file. So you can see the story.

What's next?

After we got access to the machine, we analyzed the attack, wrote a script that automatically bans bots. It is worth noting that when the number of blocked IP is more than a few hundred, the iptables option does not work . Because iptables is extremely inefficient with large lists. iptables needs to be used in conjunction with ipset , which is just designed to store and support large IP lists for iptables. You can read about the details in this article .

We hope that our experience will help in the fight against attacks.

Thanks for attention.

Actually scripts:

blockOnHighLoad.sh

 #!/bin/bash state=normal i=0 while [ 1 ]; do up=`uptime` loadavg=`echo $up | sed 's/.*average: //' | sed 's/,.\+//' | sed 's/\..\+//'` i=$(($i+1)) if (( $i > 60 )); then echo echo -n `date` " " i=0 fi if (( $loadavg > $2 ));then if [ "$state" == "normal" ];then state=high echo echo `date` HIGH $3 $3 else echo -n ${loadavg}H fi else if [ "$state" == "high" ];then state=normal echo echo `date` Normal $1 $1 else echo -n ${loadavg}. fi fi sleep 1 done 

blockAllExcept.sh (Script taken from here )

 #!/bin/sh # Minimal emergency firewall (block everything except SSH). iptables -P INPUT ACCEPT iptables -P OUTPUT ACCEPT iptables -P FORWARD ACCEPT iptables -F iptables -X iptables -A INPUT -p tcp -m tcp --dport $1 -j ACCEPT iptables -P INPUT DROP iptables -A OUTPUT -p tcp -m tcp --sport $1 -j ACCEPT iptables -P OUTPUT DROP