New Googlebot User Agent for Smartphones

Level of training webmaster: high

Google indexes content optimized for regular mobile phones and smartphones with a wide range of features and capabilities using various search robots. But we noticed that, since they are both called Googlebot-Mobile , this is often confusing. For example, webmasters, intending to prohibit scanning and indexing a site for simple mobile phones with limited technical capabilities, by mistake prohibit scanning a site for all devices. Of course, all this has a negative effect on the traffic of the website.

New Googlebot for smartphones

To make it easier for webmasters to understand the settings, in 3-4 weeks we will rename the user agent in our search robot for smartphones. The name of Googlebot-Mobile will be replaced with Googlebot with the addition of the word mobile in the user agent string. You can compare the differences below.

Googlebot is a new user agent for smartphones:

 Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible;<b>Googlebot</b>/2.1; +http://www.google.com/bot.html) 

Googlebot-Mobile is an old smartphone user agent that will soon be replaced:

 Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html) 

These changes only apply to Googlebot-Mobile for smartphones. The usual Googlebot robot will have the same user agent, and the two remaining Googlebot-Mobile search robots will still have the middle-class phone numbers in the user agent line. How it looks, see below.

User agent on a regular Googlebot search robot:

 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 

User agents for two Googlebot-Mobile search robots for middle-class phones:

• SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
• DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

To check if these search engine robots can index your site, you can use the " View as Googlebot " feature in Webmaster Tools . A complete list of our search robots is published in the Help Center .
')

Crawl and Index

Please note that after changing the search agent, the new Googlebot for smartphones will follow the rules in robots.txt , the robots meta tag and the HTTP headers for Googlebot and not Googlebot-Mobile . Here, for example, is a rule in robots.txt that prohibits crawling a site both to a regular Googlebot robot and to a new Googlebot robot for smartphones:

 User-agent: Googlebot Disallow: / 

And such a rule in robots.txt prohibits scanning of Google search robots for simple mobile phones:

 User-agent: Googlebot-Mobile Disallow: / 

According to our calculations, this update will affect less than 0.001% of web pages, but it will allow webmasters to more accurately control the scanning and indexing of their content. If you have any questions, you can:

• Read the recommendations on optimizing sites for smartphones ;
• Learn how to manage the crawling and indexing permissions of Googlebot search robots
• ask a question here in the comments, on the webmaster forum, in the webmaster community, or take part in regular webcasting with webmasters .