Scan and index pages in multiple languages

Many large sites every day attract visitors not only in the country from which they come. Localized versions are made for foreign visitors: some are translated better, some are worse, on some information as much as on the site’s native language, on others - quite a bit. However, it is not easy for Google search engine to index content on multilingual sites.



Often, the choice of which language to send information by default is made either manually, saving information about the selected language in cookies, or automatically: according to the language settings of the browser, operating system, using geolocation and determining the country / city by IP. The Googlebot crawling robot is not a user, the language itself will not select, and the automatic detection system will not recognize it and will not display all languages ​​at once, so it took some effort to correctly index the content.

')

As it was before

Previously, Googlebot had a US IP address, and sent requests without an Accept-Language header. In this situation, most sites silently gave him their English version, and on multilingual resources there could be problems with content indexing.

For example, when entering the Russian site, where users from abroad always provided content in English, the robot could not get the main content of the site in Russian. As a result, in the search results in Russia, the pages of such a site could be displayed lower than they deserved, or not displayed at all, if the setting “display results in Russian” was set.

How Googlebot works now

The algorithm of the robot has changed: if materials can be selected depending on the user's location and language settings on his device, Googlebot will process such pages using new scanning configurations that take into account location and language. Here are the solutions that are used:

• Sending a request from other IP addresses that do not belong to the United States in addition to those used so far from the United States.
  
• The use of Accept-Language headers in the request, which allows Googlebot to navigate to different language versions of the page and scan them.
  

New scan configurations are activated automatically, so your pages may be processed differently by Google’s robots and displayed in search results, even if you haven’t changed the settings of the server or content management system.

What should be done to ensure that a multilingual website appears correctly?

Want to provide the most efficient indexing and ranking of your content and make the site more user-friendly? We still recommend using different URLs for language versions and selecting the necessary options using the attributes rel = "alternate" hreflang = "x" .

If you have any questions or suggestions, we are waiting for you on our webmaster help forum or in the webmaster community on Google+: g.co/ruwebmasters .

Reference materials

Multiregional and multilingual sites (Help Center Section)
Six tips for building websites for a multilingual audience from Google Web Studio