A simple and effective method to reflect http DDoS from 50mbit using nginx and iptables
Hello, Habr!
I offer you a simple and at the same time effective method of dealing with http DDoS. Based on the Xeon 2.5GHz / 4Gb RAM / SAS server, you can repel an attack up to about 300 Mbps (the value is obtained by an extrapolation method).
The system parameters are fine tuned. So the north will be able to withstand more connections from the botnet than the channel to the server can skip.
Fight Http DDoS on a dedicated server or VPS. The maximum possible power to deter DDoS attacks is limited by the physical capabilities of the server and the bandwidth of the channel.
')
Your site will be correctly indexed during the attack, which will allow you to maintain your position in the search engines. Especially true for sites with large SEO budgets.
At the time of the attack will have to abandon some of the services of your site. You may have to expand the channel bandwidth, move the site to a more powerful server. Efficiency is achieved by maximizing the scalability of the system. Provides rapid build-up of hardware resources with an increase in attack power.
I will talk about the application of the method and the results achieved, based on a real case of the fight against http DDoS attack.
I had two Xeon 2.5GHz / 4Gb RAM / SAS servers, the first for PHP, the second for the database. All settings were made on the first server. OC - Debian 4, the site was with ~ 60k traffic. The frontend was nginx. The system kernel was configured by default. The standard ban tool for ip - iptables in a particular case has coped with a botnet attack of up to 7K in size.
In the case of a more powerful attack will have to install ipset.
The IP address allocated under DOS will stop responding to any requests (ping, http, ssh), while the rest of the IP server will continue to work properly. If the server has several IPs then the site under DOS will fall, the work of other sites on the server and ssh will not be broken.
By default, the Debian OS and other OSes are not able to support the huge number of connections created by the botnet. You need to make changes to the kernel settings to strengthen the TCP / IP stack. I will not elaborate on the configuration of the kernel, I will give only an example of such a configuration.
Similarly, you can read about the parameters in the documentation, for example debian.telenet.ru/doc/sysctl.conf , and better search through google.com for the latest articles on this topic.
Carefully change the kernel configuration and restart the server ...
So. Our system is able to withstand the onslaught of bots. But celebrating the victory is still very early. Due to the huge number of connections, PHP and database processes completely “eat up” memory and processor resources, so the load average value exceeds 100 points.
Necessary to cut parasitic connections
The anti-dos administrator, whom I addressed with the problem, suggested a method for finding bots with the netstat command. In the process of applying this method, I noticed several significant flaws. Consider them in detail:
1. Creating a blacklist takes a lot of time, which does not allow us to frequently update blacklist
2. Effective bots search is possible only when the web server is stopped. At this time, the site is not available to customers and there is a risk of improper site indexing by search engines.
3. The blacklist can get IP search robots, which is unacceptable
Realizing the ineffectiveness of the proposed method, I began to create my own search method and ban bots which should
1. ensure the continued stable operation of the web server (site)
2. guarantees the least probability in the blacklist of search robots
Xeon 2.5GHz / 4Gb RAM / SAS DoS server with GET / HTTP / 1.1 requests .
<...> *
Based on the experiment, we can draw the following conclusions:
a) You can completely abandon filtering if there is enough channel capacity and no inbound / outbound traffic ratios.
Your site will be available to customers at the cost of huge parasitic traffic.
Frivolous solution to completely abandon filtering. Attackers can increase the power of DoS so that the gigabit channel will fall.
b) When we ban absolutely all bots, the parasitic traffic from the botnet will be only 5 Mbit / s. Banning all bots is also impossible, it will take too many resources. In addition, the probability of a ban of search robots is high.
It is also necessary to pay attention to the fact that outgoing traffic with the latter case exceeded 100 Mbit / s. This means that the server connected to the 100 Mbps port will become very difficult to access via ssh due to the full load of the channel. To avoid such trouble, I recommend setting the return of empty HTML or return 444 in nginx before the bot filtering setup is complete.
In this case, the server is attacked by requests : "GET / HTTP / 1.1" .
We make an assumption that good customers make no more than 2 simultaneous requests to the main page of the site. We consider clients who open more than 3 simultaneous connections by attacking bots and ban their IP addresses on the firewall.
The assumption was confirmed experimentally. Based on the analysis of the http http requests per day from 120,000 IP addresses, only with 19 IPs, more than 2 simultaneous requests were made.
To implement the search bots create a special request processing.
request: “GET / HTTP / 1.1” in nginx.
IP addresses from which more than 3 simultaneous connections were opened will be recorded in error.log with a message limiting connections by zone. Based on the error log, we can build the blacklist ip of the attacking botnet.
It is important to notice. IPtables are not suitable for filtering a large number of addresses. With the number of chains> 2K iptables, the ksoftirqd process begins to consume 100% of the CPU, which leads to an extremely high server load. The problem is solved by installing ipset or reducing the number of rules in iptables.
In this case, the ipset installation was postponed in case of emergency. The server did not have a built-in KVM and rebuilding the kernel was risky.
Let's start creating a blacklist. In the ban, we will place only the most aggressive bots, so as not to overload iptables.
Add the script to the cron with a frequency of several minutes. We select the frequency by experience. I did every 5 minutes.
As a result, iptables will be updated with new bots.
Filtration scheme
This method has provided stable access for clients to the site. Correct indexing in the PS was confirmed by the fact that the site retained its position in the issue. Server load did not go beyond reasonable limits la no more than 6-7 points. Outgoing traffic from the server did not exceed 100 Mbps. To repel an attack> 7K botnet, iptables is quite enough.
DDoS as a natural disaster and avoid damage is impossible.
Some customers, during the failure of your service, will go to competitors.
You will have to incur some processing costs for programmers, administrators or the purchase of additional equipment.
Your resource is actively promoted in PS (yandex, google) which means that the risk of incorrect indexation is critical and, as a result, the failure of positions in the issue.
The main task is to minimize the damage from DDoS.
In my case, the DDoS attack stopped the next day after the start of filtering. The DoS customer was not willing to spend more money to increase the attack.
In most cases, DDoS is a competitive online weapon. Clients are almost instantly able to go to your competitors if your resource will malfunction.
I believe that the fight against DDoS is not in the bots bath, but in creating conditions in which your total damage from an attack is comparable to the costs of its initiators. The customer must spend, for example, 50 000 rubles. in order to cause you damage in 50 000 rub., competitors are not economically profitable to organize such attacks.
The method described in this article is not a panacea, it is only part of a set of measures to reflect DoS. The development plan for a large service should take into account risks and suggest measures to reduce the negative effects of attacks.
I hope my article will be useful to the community of developers and administrators of web applications.
___
* I removed a paragraph about 300Mbps from the text, because He reasonably causes complaints.
“Over 300 Mbit / s, we will“ push ”to the limit ...” - true for HDD video / audio output, i.e. "Heavy" files. For HTML files this statement is unfair.
Paragraph deleted text:
“According to the results of the experiment, it is clear that the server is able to withstand an increase in attack of approximately 300 Mbit / s. Over 300 Mbit / s, we will “rest” in the limit of random read SAS disks. So, we have a good margin of safety and a high probability of effectively repelling an attack while maintaining the availability of our web services to customers. ”
I offer you a simple and at the same time effective method of dealing with http DDoS. Based on the Xeon 2.5GHz / 4Gb RAM / SAS server, you can repel an attack up to about 300 Mbps (the value is obtained by an extrapolation method).
Implementation method
The system parameters are fine tuned. So the north will be able to withstand more connections from the botnet than the channel to the server can skip.
Application area
Fight Http DDoS on a dedicated server or VPS. The maximum possible power to deter DDoS attacks is limited by the physical capabilities of the server and the bandwidth of the channel.
')
SEO under DDoS
Your site will be correctly indexed during the attack, which will allow you to maintain your position in the search engines. Especially true for sites with large SEO budgets.
Cost and effectiveness
At the time of the attack will have to abandon some of the services of your site. You may have to expand the channel bandwidth, move the site to a more powerful server. Efficiency is achieved by maximizing the scalability of the system. Provides rapid build-up of hardware resources with an increase in attack power.
Method Description
I will talk about the application of the method and the results achieved, based on a real case of the fight against http DDoS attack.
I had two Xeon 2.5GHz / 4Gb RAM / SAS servers, the first for PHP, the second for the database. All settings were made on the first server. OC - Debian 4, the site was with ~ 60k traffic. The frontend was nginx. The system kernel was configured by default. The standard ban tool for ip - iptables in a particular case has coped with a botnet attack of up to 7K in size.
In the case of a more powerful attack will have to install ipset.
The history of the fight against DDoS
The first day. Network stack overflow
The IP address allocated under DOS will stop responding to any requests (ping, http, ssh), while the rest of the IP server will continue to work properly. If the server has several IPs then the site under DOS will fall, the work of other sites on the server and ssh will not be broken.
By default, the Debian OS and other OSes are not able to support the huge number of connections created by the botnet. You need to make changes to the kernel settings to strengthen the TCP / IP stack. I will not elaborate on the configuration of the kernel, I will give only an example of such a configuration.
net.ipv4.conf.all.accept_redirects = 0
net.ipv4.conf.eth0.accept_redirects = 0
net.ipv4.conf.default.accept_redirects = 0
net.core.rmem_max = 996777216
net.core.wmem_max = 996777216
net.ipv4.tcp_rmem = 4096 87380 4194304
net.ipv4.tcp_mem= 786432 1048576 996777216
net.ipv4.tcp_wmem = 4096 87380 4194304
net.ipv4.tcp_max_orphans = 2255360
net.core.netdev_max_backlog = 10000
net.ipv4.tcp_fin_timeout = 10
net.ipv4.tcp_keepalive_intvl = 15
net.ipv4.tcp_max_syn_backlog = 2048
net.ipv4.tcp_synack_retries = 1
kernel.msgmnb = 65536
kernel.msgmax = 65536
kernel.shmmax = 494967295
kernel.shmall = 268435456
net.core.somaxconn= 16096Similarly, you can read about the parameters in the documentation, for example debian.telenet.ru/doc/sysctl.conf , and better search through google.com for the latest articles on this topic.
Carefully change the kernel configuration and restart the server ...
So. Our system is able to withstand the onslaught of bots. But celebrating the victory is still very early. Due to the huge number of connections, PHP and database processes completely “eat up” memory and processor resources, so the load average value exceeds 100 points.
Necessary to cut parasitic connections
Disadvantages of bots search with the netstat command
The anti-dos administrator, whom I addressed with the problem, suggested a method for finding bots with the netstat command. In the process of applying this method, I noticed several significant flaws. Consider them in detail:
1. Creating a blacklist takes a lot of time, which does not allow us to frequently update blacklist
2. Effective bots search is possible only when the web server is stopped. At this time, the site is not available to customers and there is a risk of improper site indexing by search engines.
3. The blacklist can get IP search robots, which is unacceptable
Realizing the ineffectiveness of the proposed method, I began to create my own search method and ban bots which should
1. ensure the continued stable operation of the web server (site)
2. guarantees the least probability in the blacklist of search robots
Second day. Possibilities of server hardware + nginx
Xeon 2.5GHz / 4Gb RAM / SAS DoS server with GET / HTTP / 1.1 requests .
- Experiment A. The web server (in this case, nginx) is stopped
Inbound traffic 6085.2 kbits / sec
Outgoing traffic 5342.1 kbits / sec - Experiment B. Nginx gives empty HTML (return 444;)
56 Mbps inbound traffic
Outgoing traffic 54 Mbit / s - Experiment B. Nginx gives HTML of about 2 Kb in size - this is a page with a small message like “we apologize for the interruptions in the work of the site”
57 Mbps inbound traffic
Outgoing traffic 353 Mbps
<...> *
Based on the experiment, we can draw the following conclusions:
a) You can completely abandon filtering if there is enough channel capacity and no inbound / outbound traffic ratios.
Your site will be available to customers at the cost of huge parasitic traffic.
Frivolous solution to completely abandon filtering. Attackers can increase the power of DoS so that the gigabit channel will fall.
b) When we ban absolutely all bots, the parasitic traffic from the botnet will be only 5 Mbit / s. Banning all bots is also impossible, it will take too many resources. In addition, the probability of a ban of search robots is high.
It is also necessary to pay attention to the fact that outgoing traffic with the latter case exceeded 100 Mbit / s. This means that the server connected to the 100 Mbps port will become very difficult to access via ssh due to the full load of the channel. To avoid such trouble, I recommend setting the return of empty HTML or return 444 in nginx before the bot filtering setup is complete.
Nginx bots search
In this case, the server is attacked by requests : "GET / HTTP / 1.1" .
We make an assumption that good customers make no more than 2 simultaneous requests to the main page of the site. We consider clients who open more than 3 simultaneous connections by attacking bots and ban their IP addresses on the firewall.
The assumption was confirmed experimentally. Based on the analysis of the http http requests per day from 120,000 IP addresses, only with 19 IPs, more than 2 simultaneous requests were made.
To implement the search bots create a special request processing.
request: “GET / HTTP / 1.1” in nginx.
error_log /var/log/nginx/error.log;
<…>
location =/ {
limit_conn one 3;
root /home/www/site.ru;
}
IP addresses from which more than 3 simultaneous connections were opened will be recorded in error.log with a message limiting connections by zone. Based on the error log, we can build the blacklist ip of the attacking botnet.
Filtering bots in iptables
It is important to notice. IPtables are not suitable for filtering a large number of addresses. With the number of chains> 2K iptables, the ksoftirqd process begins to consume 100% of the CPU, which leads to an extremely high server load. The problem is solved by installing ipset or reducing the number of rules in iptables.
In this case, the ipset installation was postponed in case of emergency. The server did not have a built-in KVM and rebuilding the kernel was risky.
Let's start creating a blacklist. In the ban, we will place only the most aggressive bots, so as not to overload iptables.
#
cat /var/log/nginx/error.log | grep "limiting connections by zone" | grep "request: \"GET / HTTP/1.1"| awk '{print $12}'| awk -F"," '{print $
1}'| sort | uniq -c | sort -nr > /tmp/botnet.blacklist
#
cat /dev/null > /tmp/iptables_ban.sh
# DROP 50
awk '{print "iptables -A INPUT -p tcp --dport 80 -s " $2 " -j DROP" }' botnet.blacklist | head -n 50 >> /tmp/iptables_ban.sh
# blacklist
bash /tmp/iptables_ban.sh
#
cat /dev/null > /home/www/nginx_log/error.log
[ ! -f /var/run/nginx.pid ] || kill -USR1 `cat /var/run/nginx.pid`Add the script to the cron with a frequency of several minutes. We select the frequency by experience. I did every 5 minutes.
*/5 * * * * /root/script/ban.shAs a result, iptables will be updated with new bots.
Filtration scheme
Third day. Total
This method has provided stable access for clients to the site. Correct indexing in the PS was confirmed by the fact that the site retained its position in the issue. Server load did not go beyond reasonable limits la no more than 6-7 points. Outgoing traffic from the server did not exceed 100 Mbps. To repel an attack> 7K botnet, iptables is quite enough.
DDoS as a natural disaster and avoid damage is impossible.
Some customers, during the failure of your service, will go to competitors.
You will have to incur some processing costs for programmers, administrators or the purchase of additional equipment.
Your resource is actively promoted in PS (yandex, google) which means that the risk of incorrect indexation is critical and, as a result, the failure of positions in the issue.
The main task is to minimize the damage from DDoS.
In my case, the DDoS attack stopped the next day after the start of filtering. The DoS customer was not willing to spend more money to increase the attack.
In most cases, DDoS is a competitive online weapon. Clients are almost instantly able to go to your competitors if your resource will malfunction.
I believe that the fight against DDoS is not in the bots bath, but in creating conditions in which your total damage from an attack is comparable to the costs of its initiators. The customer must spend, for example, 50 000 rubles. in order to cause you damage in 50 000 rub., competitors are not economically profitable to organize such attacks.
The method described in this article is not a panacea, it is only part of a set of measures to reflect DoS. The development plan for a large service should take into account risks and suggest measures to reduce the negative effects of attacks.
I hope my article will be useful to the community of developers and administrators of web applications.
___
* I removed a paragraph about 300Mbps from the text, because He reasonably causes complaints.
“Over 300 Mbit / s, we will“ push ”to the limit ...” - true for HDD video / audio output, i.e. "Heavy" files. For HTML files this statement is unfair.
Paragraph deleted text:
“According to the results of the experiment, it is clear that the server is able to withstand an increase in attack of approximately 300 Mbit / s. Over 300 Mbit / s, we will “rest” in the limit of random read SAS disks. So, we have a good margin of safety and a high probability of effectively repelling an attack while maintaining the availability of our web services to customers. ”
Source: https://habr.com/ru/post/84172/
All Articles