How to stop Rambler?
The most outdated search engine of the Russian Internet seems to have loved us in earnest.
Rambler requests documents from habrakhabra with a speed of up to 170 requests per second, which does not suit us somewhat and according to our observations is an absolute record among all indexers visiting our project.
From the technical support service of the search engine received a few incoherent paranoid answers, advising you to read the page describing the robots.txt file or “put a delay on the answer to the robot”.
')
Apparently, Rambler does not support the Crawl-delay directive and finding a solution to the problem without blocking access to Rambler will not be as easy as we would like (nginx in a step from solving this problem for the desired user-agent value).
I wonder if there are those among the readers who can boast of even greater degree of attention from the Rambler?
Rambler requests documents from habrakhabra with a speed of up to 170 requests per second, which does not suit us somewhat and according to our observations is an absolute record among all indexers visiting our project.
From the technical support service of the search engine received a few incoherent paranoid answers, advising you to read the page describing the robots.txt file or “put a delay on the answer to the robot”.
')
Apparently, Rambler does not support the Crawl-delay directive and finding a solution to the problem without blocking access to Rambler will not be as easy as we would like (nginx in a step from solving this problem for the desired user-agent value).
I wonder if there are those among the readers who can boast of even greater degree of attention from the Rambler?
Source: https://habr.com/ru/post/38171/
All Articles