DDoS from an unexpected direction or “are PS bots afraid?”
I want to tell an interesting story about the interaction with search engines, and in particular with the bots of the great and powerful Yandex.
Preamble. I have a dedicated server on which the order of several dozen sites is spinning. There were never any problems, the machine is vigorous, new. A few days ago I received a report that everything was “stuck”. I had to do a remote reboot. I later noticed that the CPU load rose to 60% from 10% and began to stay at this level. Of course, I was wary, but it was not enough.
And yesterday everything hangs completely. Reboot - progruz and again complete calm. All sites go out of access. After monitoring the processes, I saw mysql loading at 99.9%. I was surprised, I went to look for the site from which such download is coming. I found, weed out other requests and saw that someone was hammering hard at 40k requests per second, overloading the poor database with “I can’t do it myself”. I start digging logs and see the subnets from which these requests come. I connect to the server, I register DROP in iptables according to the mask / 24, and everything is normalized.
I noticed that the ddaser sends the Yandex-bot identification. Slightly this surprised, but you never know. I decided to check, and it turned out that the two subnets that I had to ban were owned by Yandex spiders.
')
After a thorough study of why and what the bots did on the site with such frenzy, it became clear that the bots had reached the unclosed filter of products and sped off an incredible pile of pages, which they began to scroll through for indexing. Why some sort of cut-off did not work for them like that - it is not clear. The site with this filter works for more than three years and there were no problems.
At the moment, the filter is closed from indexing via htaccess, the bots are unbanned, and I am waiting for a response from Yandex support.
And I understand, if I were more prudent, there would be no such problems. But still.
Thank.
Preamble. I have a dedicated server on which the order of several dozen sites is spinning. There were never any problems, the machine is vigorous, new. A few days ago I received a report that everything was “stuck”. I had to do a remote reboot. I later noticed that the CPU load rose to 60% from 10% and began to stay at this level. Of course, I was wary, but it was not enough.
And yesterday everything hangs completely. Reboot - progruz and again complete calm. All sites go out of access. After monitoring the processes, I saw mysql loading at 99.9%. I was surprised, I went to look for the site from which such download is coming. I found, weed out other requests and saw that someone was hammering hard at 40k requests per second, overloading the poor database with “I can’t do it myself”. I start digging logs and see the subnets from which these requests come. I connect to the server, I register DROP in iptables according to the mask / 24, and everything is normalized.
I noticed that the ddaser sends the Yandex-bot identification. Slightly this surprised, but you never know. I decided to check, and it turned out that the two subnets that I had to ban were owned by Yandex spiders.
')
After a thorough study of why and what the bots did on the site with such frenzy, it became clear that the bots had reached the unclosed filter of products and sped off an incredible pile of pages, which they began to scroll through for indexing. Why some sort of cut-off did not work for them like that - it is not clear. The site with this filter works for more than three years and there were no problems.
At the moment, the filter is closed from indexing via htaccess, the bots are unbanned, and I am waiting for a response from Yandex support.
A small piece of logs
[29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & show = 50 & page = 7 & page = 5 & show = 10 & page = 4 & page = 6 HTTP / 1.1" 200 102 "-" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
87.250.244.29 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & show = 50 & page = 7 & page = 5 & show = 10 & page = 4 & page = 1 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
87.250.244.16 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & show = 50 & page = 7 & page = 5 & show = 10 & page = 4 & page = 2 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
141.8.141.140 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 111 & show = 50 & page = 1 & page = 23 & show = 30 & page = 37 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
87.250.244.16 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 111 & show = 50 & page = 1 & page = 23 & show = 30 & page = 26 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
141.8.141.140 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 111 & show = 50 & page = 1 & page = 23 & show = 30 & page = 25 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
141.8.141.139 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 111 & show = 50 & page = 1 & page = 23 & show = 30 & page = 24 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
87.250.244.29 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 111 & show = 50 & page = 1 & page = 23 & show = 30 & page = 20 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
87.250.244.16 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 111 & show = 50 & page = 1 & page = 23 & show = 30 & page = 21 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
141.8.141.140 - - [29 / Oct / 2015: 13: 57: 10] “GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 5 & show = 10 & page = 3 & show = 50 & page = 1 & page = 3 & page = 6 & page = 23 HTTP / 1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
141.8.141.139 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 1 & page = 3 & show = 30 & page = 1 & page = 3 & page = 37 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
87.250.244.29 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 5 & page = 4 & page = 1 & page = 2 & page = 111 HTTP / 1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
87.76.13.189 - - [29 / Oct / 2015: 13: 57: 10] “GET / HTTP / 1.1” 200 102 "-" "Mozilla / 5.0 (Windows NT 6.3; WOW64) AppleWebKit / 537.36 (KHTML, like Gecko) Chrome / 46.0.2490.80 Safari / 537.36 "
87.250.244.16 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 8 & page = 7 & page = 8 & page = 6 & page = 4 & page = 5 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
141.8.141.140 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 8 & page = 7 & page = 8 & page = 6 & page = 4 & page = 7 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
141.8.141.139 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 8 & page = 7 & page = 8 & page = 6 & page = 4 & page = 6 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
87.250.244.29 - - [29 / Oct / 2015: 13: 57: 10] “GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 8 & page = 7 & page = 8 & page = 6 & page = 4 & page = 1 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
87.250.244.16 - - [29 / Oct / 2015: 13: 57: 10] “GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 8 & page = 7 & page = 8 & page = 6 & page = 4 & page = 2 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
141.8.141.138 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 4 & show = 50 & show = 10 & show = 30 & page = 37 HTTP / 1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
141.8.141.143 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 1 & show = 50 & page = 4 & show = 30 & page = 37 HTTP / 1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
141.8.141.137 - - [29 / Oct / 2015: 13: 57: 10] “GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & show = 30 & page = 1 & page = 3 & page = 2 & page = 3 & page = 1 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
141.8.141.132 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & show = 30 & page = 1 & page = 3 & page = 2 & page = 3 & page = 6 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
87.250.244.15 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & show = 30 & page = 1 & page = 3 & page = 2 & page = 3 & page = 4 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
87.250.244.38 - - [29 / Oct / 2015: 13: 57: 10] “GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & show = 30 & page = 1 & page = 3 & page = 2 & page = 3 & page = 5 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
87.250.244.14 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 7 & page = 9 & show = 30 & page = 11 & show = 50 HTTP / 1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0;
87.250.244.29 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & show = 50 & page = 7 & page = 5 & show = 10 & page = 4 & page = 1 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
87.250.244.16 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & show = 50 & page = 7 & page = 5 & show = 10 & page = 4 & page = 2 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
141.8.141.140 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 111 & show = 50 & page = 1 & page = 23 & show = 30 & page = 37 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
87.250.244.16 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 111 & show = 50 & page = 1 & page = 23 & show = 30 & page = 26 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
141.8.141.140 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 111 & show = 50 & page = 1 & page = 23 & show = 30 & page = 25 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
141.8.141.139 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 111 & show = 50 & page = 1 & page = 23 & show = 30 & page = 24 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
87.250.244.29 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 111 & show = 50 & page = 1 & page = 23 & show = 30 & page = 20 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
87.250.244.16 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 111 & show = 50 & page = 1 & page = 23 & show = 30 & page = 21 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
141.8.141.140 - - [29 / Oct / 2015: 13: 57: 10] “GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 5 & show = 10 & page = 3 & show = 50 & page = 1 & page = 3 & page = 6 & page = 23 HTTP / 1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
141.8.141.139 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 1 & page = 3 & show = 30 & page = 1 & page = 3 & page = 37 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
87.250.244.29 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 5 & page = 4 & page = 1 & page = 2 & page = 111 HTTP / 1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
87.76.13.189 - - [29 / Oct / 2015: 13: 57: 10] “GET / HTTP / 1.1” 200 102 "-" "Mozilla / 5.0 (Windows NT 6.3; WOW64) AppleWebKit / 537.36 (KHTML, like Gecko) Chrome / 46.0.2490.80 Safari / 537.36 "
87.250.244.16 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 8 & page = 7 & page = 8 & page = 6 & page = 4 & page = 5 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
141.8.141.140 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 8 & page = 7 & page = 8 & page = 6 & page = 4 & page = 7 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
141.8.141.139 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 8 & page = 7 & page = 8 & page = 6 & page = 4 & page = 6 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
87.250.244.29 - - [29 / Oct / 2015: 13: 57: 10] “GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 8 & page = 7 & page = 8 & page = 6 & page = 4 & page = 1 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
87.250.244.16 - - [29 / Oct / 2015: 13: 57: 10] “GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 8 & page = 7 & page = 8 & page = 6 & page = 4 & page = 2 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
141.8.141.138 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 4 & show = 50 & show = 10 & show = 30 & page = 37 HTTP / 1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
141.8.141.143 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 1 & show = 50 & page = 4 & show = 30 & page = 37 HTTP / 1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
141.8.141.137 - - [29 / Oct / 2015: 13: 57: 10] “GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & show = 30 & page = 1 & page = 3 & page = 2 & page = 3 & page = 1 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
141.8.141.132 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & show = 30 & page = 1 & page = 3 & page = 2 & page = 3 & page = 6 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
87.250.244.15 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & show = 30 & page = 1 & page = 3 & page = 2 & page = 3 & page = 4 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
87.250.244.38 - - [29 / Oct / 2015: 13: 57: 10] “GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & show = 30 & page = 1 & page = 3 & page = 2 & page = 3 & page = 5 HTTP /1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0; + http: //yandex.com/bots) "
87.250.244.14 - - [29 / Oct / 2015: 13: 57: 10] "GET / catalog / kotli /? 176 & filter = 1 & fldX = 0 & page = 8 & page = 6 & show = 10 & page = 7 & page = 9 & show = 30 & page = 11 & show = 50 HTTP / 1.1 "200 102" - "" Mozilla / 5.0 (compatible; YandexBot / 3.0;
And I understand, if I were more prudent, there would be no such problems. But still.
Thank.
Source: https://habr.com/ru/post/270297/
All Articles