DDoS-attack on Habrahabr: post-mortem
On July 26, Pas addressed us with the most unpleasant news: Habr was attacked again. We, of course, advised to switch to us.
While the DNS was updated, we made the initial blackbox evaluation of the battlefield:
So, the server is responsible for a reasonable time, but the connection is not established. Apparently, the attack goes to the TCP protocol or application layer. After the recent 50-gigabit attack, we have a quarantine for difficult customers, but in this case it was decided not to use it - the traffic is probably a tiny amount.
At half past seven in the evening, the DNS switched.
')
This graphic shows a fox. In the case of a working Web resource, the volume of outgoing traffic exceeds the amount of incoming traffic by an order of magnitude. If they differ by several tens of percent - trouble.
The last time Habr stood under our protection, hmm, for a long time. There is virtually no accumulated behavior history of legitimate users, you have to control the filters manually. First we include a rough analysis of the TCP connection automaton.
It turns out that they are attacking with a high rate of connections and requests that are timed out. Connection tracking, by itself, flew to hell, the base was also bad. Bots are blocked in large quantities.
Habraserver begins to be more alive than dead. But there are garbage requests, it can be seen.
While the court and the case, accumulated a critical mass in the classifier. We start the analysis of transitions. Immediately in the "red zone" flies a cloud of IP-addresses that perform these requests here:
The query is always the same: GET / search? Q = intel. Referrers are different: Google, Ask.com and other search engines. The authors of these illegitimate requests are blocked.
Do not worry: even if at that time you came from Google to the Habr search page with the query “intel”, it is unlikely that you were immediately banned. Calculation of the legitimacy of the user is based on a variety of requests.
Please note: as soon as the parasitic traffic was filtered, the traffic generated by legitimate users (the site would have earned) slightly increased a little and outbound traffic instantly awakened. Within an hour after the protection was turned on “from scratch,” the site came to a fully operational state.
We surviveled HTTP QRATOR 503!
At this point, the attack basically ceased. It makes no sense to keep parasitic traffic on a site that works in spite of, it is better to make money on someone else.
Let us estimate the attack power. The following striking components are available:
- number of connections;
- the number of requests to the database.
Accordingly, it is pointless to estimate the volume of incoming traffic or the number of attacking IP addresses, the main thing is the number of open connections per time unit. At the peak of the attack, about 9000 requests were made per second, which, in fact, is the attack power. Each request was executed in a separate TCP connection. Note that the default Linux value of CONNTRACK_MAX (assuming that there is more than 1 GB of RAM on the server) would thus be exhausted after 7.29 attacks; plus load on the base.
The trouble, as you know, does not come alone; in any case, not in this area. At 0:20, the attackers came again, and the botnet volume was slightly more than at 19:30. But we were ready for this, and the attack was automatically canceled. Attack relapse is a very common thing, sometimes the organizers specifically monitor the behavior of the site and activate the botnet as soon as its participants begin to “let go”. It was easier here, the repeated attack stopped as the first, as soon as all the bots were blocked.
EDIT 14:52 Literally, with only minimal losses, the third wave of the attack ended. About 4500 attacking bots, 4000 requests at the peak. Infected cars are now not only interested in Intel, but also Apple and Google. We continue to monitor developments.
While the DNS was updated, we made the initial blackbox evaluation of the battlefield:
ximaera@endeavour:14#487:~$ nc habrahabr.ru 80 <<EOF
> GET / HTTP/1.1
> Host: habrahabr.ru
>
> EOF
^C
real 0m19.020s
user 0m0.000s
sys 0m0.024s
ximaera@endeavour:14#488:~$ ping habrahabr.ru
64 bytes from ***: icmp_req=1 ttl=54 time=53.5 ms
64 bytes from ***: icmp_req=2 ttl=54 time=53.1 ms
64 bytes from ***: icmp_req=3 ttl=54 time=53.1 ms
^C
--- *** ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2001ms
rtt min/avg/max/mdev = 53.126/53.300/53.598/0.340 ms
ximaera@endeavour:14#489:~$
So, the server is responsible for a reasonable time, but the connection is not established. Apparently, the attack goes to the TCP protocol or application layer. After the recent 50-gigabit attack, we have a quarantine for difficult customers, but in this case it was decided not to use it - the traffic is probably a tiny amount.
At half past seven in the evening, the DNS switched.
')
This graphic shows a fox. In the case of a working Web resource, the volume of outgoing traffic exceeds the amount of incoming traffic by an order of magnitude. If they differ by several tens of percent - trouble.
The last time Habr stood under our protection, hmm, for a long time. There is virtually no accumulated behavior history of legitimate users, you have to control the filters manually. First we include a rough analysis of the TCP connection automaton.
It turns out that they are attacking with a high rate of connections and requests that are timed out. Connection tracking, by itself, flew to hell, the base was also bad. Bots are blocked in large quantities.
Habraserver begins to be more alive than dead. But there are garbage requests, it can be seen.
While the court and the case, accumulated a critical mass in the classifier. We start the analysis of transitions. Immediately in the "red zone" flies a cloud of IP-addresses that perform these requests here:
178.120.56.144 - - [26/Jul/2011:20:00:59 +0400] "GET /search/?q=intel HTTP/1.0" 200 14443 "http://www.live.com/" "Mozilla/5.0 (compatible; MSIE 7.0; Windows NT 5.0)" [habrahabr.ru]
85.173.219.240 - - [26/Jul/2011:20:00:59 +0400] "GET /search/?q=intel HTTP/1.0" 200 14443 "http://www.alexa.com/" "Mozilla/5.0 (compatible; MSIE 7.0; Windows NT 6.0)" [habrahabr.ru]
178.94.52.22 - - [26/Jul/2011:20:00:59 +0400] "GET /search/?q=intel HTTP/1.0" 200 14443 "http://www.google.com/" "Mozilla/5.0 (compatible; MSIE 7.0; Windows NT 6.1)" [habrahabr.ru]
46.36.130.136 - - [26/Jul/2011:20:00:59 +0400] "GET /search/?q=intel HTTP/1.0" 200 14443 "http://www.live.com/" "Opera/7.51 (Windows NT 5.2; U) [en]" [habrahabr.ru]
46.7.52.14 - - [26/Jul/2011:20:00:59 +0400] "GET /search/?q=intel HTTP/1.0" 200 14443 "http://www.live.com/" "Mozilla/5.0 (compatible; MSIE 7.0; Windows NT 5.2)" [habrahabr.ru]
The query is always the same: GET / search? Q = intel. Referrers are different: Google, Ask.com and other search engines. The authors of these illegitimate requests are blocked.
Do not worry: even if at that time you came from Google to the Habr search page with the query “intel”, it is unlikely that you were immediately banned. Calculation of the legitimacy of the user is based on a variety of requests.
Please note: as soon as the parasitic traffic was filtered, the traffic generated by legitimate users (the site would have earned) slightly increased a little and outbound traffic instantly awakened. Within an hour after the protection was turned on “from scratch,” the site came to a fully operational state.
We surviveled HTTP QRATOR 503!
At this point, the attack basically ceased. It makes no sense to keep parasitic traffic on a site that works in spite of, it is better to make money on someone else.
Let us estimate the attack power. The following striking components are available:
- number of connections;
- the number of requests to the database.
Accordingly, it is pointless to estimate the volume of incoming traffic or the number of attacking IP addresses, the main thing is the number of open connections per time unit. At the peak of the attack, about 9000 requests were made per second, which, in fact, is the attack power. Each request was executed in a separate TCP connection. Note that the default Linux value of CONNTRACK_MAX (assuming that there is more than 1 GB of RAM on the server) would thus be exhausted after 7.29 attacks; plus load on the base.
The trouble, as you know, does not come alone; in any case, not in this area. At 0:20, the attackers came again, and the botnet volume was slightly more than at 19:30. But we were ready for this, and the attack was automatically canceled. Attack relapse is a very common thing, sometimes the organizers specifically monitor the behavior of the site and activate the botnet as soon as its participants begin to “let go”. It was easier here, the repeated attack stopped as the first, as soon as all the bots were blocked.
EDIT 14:52 Literally, with only minimal losses, the third wave of the attack ended. About 4500 attacking bots, 4000 requests at the peak. Infected cars are now not only interested in Intel, but also Apple and Google. We continue to monitor developments.
Source: https://habr.com/ru/post/125018/
All Articles