New Yahoo Crawler
Following Google's upgrade, Yahoo search engine also updated its search crawler. Here the updates are more extensive, the new crawler was named Yahoo! Slurp 3.0 and the same user-agent status (Yahoo! Slurp / 3.0). The new robot user will not react to the commands of the old user-agent “Slurp / 2.0”, but it will respond to the general short appeal “Slurp”, according to a corporate blog.
As the name implies, this is a third-generation crawler. However, the company does not seek to disclose information about what its specific differences from the previous version. It only warns all analysts who track the behavior of search bots that they make the appropriate adjustments, since Slurp 3.0 will behave differently. In particular, the program will run from a much smaller set of IP addresses, although all of them will still correspond to the domain crawl.yahoo.net (all the technical information here ). For accurate identification of the crawler, the company strongly recommends using the DNS reverse lookup method, because the old IP addresses will soon disappear from the logs.
As the name implies, this is a third-generation crawler. However, the company does not seek to disclose information about what its specific differences from the previous version. It only warns all analysts who track the behavior of search bots that they make the appropriate adjustments, since Slurp 3.0 will behave differently. In particular, the program will run from a much smaller set of IP addresses, although all of them will still correspond to the domain crawl.yahoo.net (all the technical information here ). For accurate identification of the crawler, the company strongly recommends using the DNS reverse lookup method, because the old IP addresses will soon disappear from the logs.
')
Source: https://habr.com/ru/post/23580/
All Articles