Google instructions to properly disable sites January 18
Wikipedia (en) , Reddit , Minecraft.net and other sites go offline tomorrow to protest SOPA. In this way, they hope to attract the attention of the audience to this bill, which actually introduces censorship on the Internet. SOPA allows US authorities to pretrially withdraw domain names, remove sites from search results and block their financial accounts, including in partner programs like Google AdSense, and also filter content at the level of ISP providers.
Google employee Pierre Far (Pierre Far) has published recommendations on how best to disable the site, so that it does not damage the resource in the Google search results.
He explained that the site will be penalized if it returns the other content to the search bot instead of the main page. But some sites are going to put a stub on the main page with information about the reasons for the boycott, the window of live video broadcast from the US House of Representatives, where they will be hearing on SOPA. That's exactly what Reddit is going to do. According to Pierre Farah, this option will lead to the loss of positions in the search results.
1. The most important advice to boycotting sites is to return the correct 503 Service Unavailable header for all URLs that participate in the boycott. Thus, Googlebot will not index the content, while the previous version of the site will remain in the cache, and the site will not lose its position in the search results.
')
2. The frequency of indexing the site will decrease if Googlebot sees a large number of 503 headers, but then gradually returns to the previous level.
3. In the current configuration, Googlebot does not index the site at all if it receives the status of 503 for robots.txt. Thus, if only part of the site is involved in the boycott, make sure that the robots.txt file does not return status 503. It is not recommended to use the "Disallow: /" instruction in robots.txt, because this will lead to a long pause in the site indexation.
In general, a Google representative recommends that you do not complicate the situation, do not change the DNS settings and do not touch robots.txt.
Although the representative Pierre Far does not directly mention this possibility, we recall that you can issue different content for the search bot (503 Service Unavailable) and for users (a stub with information about SOPA). Although there is a risk that the site will be fined for cloaking, but there are many cases where Google does not penalize it. For example, the same NY Times has long been giving people and robots different versions of their site.
We add that the instruction of Pierre Farah will not lose its relevance, even if Google joins the boycott on January 18. Unfortunately, it is unlikely a commercial company will be able to go for it, despite all the desire, because it has legal obligations to its customers. But even in the case of a boycott of Google, it makes no sense to disable the backend and search bot.
Google employee Pierre Far (Pierre Far) has published recommendations on how best to disable the site, so that it does not damage the resource in the Google search results.
He explained that the site will be penalized if it returns the other content to the search bot instead of the main page. But some sites are going to put a stub on the main page with information about the reasons for the boycott, the window of live video broadcast from the US House of Representatives, where they will be hearing on SOPA. That's exactly what Reddit is going to do. According to Pierre Farah, this option will lead to the loss of positions in the search results.
1. The most important advice to boycotting sites is to return the correct 503 Service Unavailable header for all URLs that participate in the boycott. Thus, Googlebot will not index the content, while the previous version of the site will remain in the cache, and the site will not lose its position in the search results.
')
2. The frequency of indexing the site will decrease if Googlebot sees a large number of 503 headers, but then gradually returns to the previous level.
3. In the current configuration, Googlebot does not index the site at all if it receives the status of 503 for robots.txt. Thus, if only part of the site is involved in the boycott, make sure that the robots.txt file does not return status 503. It is not recommended to use the "Disallow: /" instruction in robots.txt, because this will lead to a long pause in the site indexation.
In general, a Google representative recommends that you do not complicate the situation, do not change the DNS settings and do not touch robots.txt.
Although the representative Pierre Far does not directly mention this possibility, we recall that you can issue different content for the search bot (503 Service Unavailable) and for users (a stub with information about SOPA). Although there is a risk that the site will be fined for cloaking, but there are many cases where Google does not penalize it. For example, the same NY Times has long been giving people and robots different versions of their site.
We add that the instruction of Pierre Farah will not lose its relevance, even if Google joins the boycott on January 18. Unfortunately, it is unlikely a commercial company will be able to go for it, despite all the desire, because it has legal obligations to its customers. But even in the case of a boycott of Google, it makes no sense to disable the backend and search bot.
Source: https://habr.com/ru/post/136433/
All Articles