“No longer means better”: counterintuitive case from Google

Two months ago, we published a selection of cases , as the speed of the site affects the conversion. We continue to understand this topic: now more in-depth information about the situation with the number of results in Google search.

Any owner of the site, online store, online service seeks to satisfy the wishes of its users. Do they want more features? Made by Do they want bigger pictures of the goods? Made by And further on the list.
')
But the constant desire to improve the functional part of the site or service has one extremely dangerous enemy - the speed of loading the site.

Many resources on the Internet cite the fact that the delay in downloading Google search results by 0.5 seconds led to a 25% drop in the number of search queries. This data was presented at a Web 2.0 conference in San Francisco in 2006.

However, as they say, “the devil is in the details,” and we decided to find out all the details of this case.

So, the story began in 2005, when a team of Google employees led by Marissa Mayer, then Google’s vice president, worked on exploring user preferences for using search engines. The study was conducted in the form of a set of surveys and in-person conversations of researchers with respondents.

One of the things that the respondents were talking about was that just 10 results (not counting ads) on the first page of the search results are not enough.

When they began to interview respondents in detail about how many search results they would like to see, the optimal number turned out to be 30. It is not clear why they didn’t say, for example, 50, but maybe they were afraid to “drown” in the information.

The respondents tested various prototypes of search results with different lengths of output, and all the results indicated that 30 is the optimal number.

Naturally, the cost of an error in Google is extremely high, and of course no one was going to “roll out” such a change at once to all users of a real service.

The team chose a test group of users, really large, with representatives of different ages, interests, countries, and so on. And this group Google gave search results, grouped by 30 results on 1 page, instead of 10 results.

At the same time, a control group was defined, with the same socio-demographic parameters, to keep track of how the usual audience behaves, which still sees 10 search results per page. This was done to cut off any factors such as seasonality, holidays, geographic influence, etc.

The experiment lasted 6 weeks. At the end of this period, the number of hits to the search from users of the experimental group that received 30 search results per page decreased by 20%.



Obviously, the introduction of such a change would have a catastrophic effect on Google’s business, since a company receives a huge share of its income from search advertising, which also depends on user experience. If the advertisement is not shown to the user, there will be no clicks on it. Reduce by 20% appeal to search, and as a result, search advertising revenue, this is billions in losses for Google.

At first, the researchers' hypothesis was that the respondents simply find everything from the first time and do not press the “next” button and do not go to the next pages.

But this was not the case - a more thorough analysis of the results showed that respondents of the experimental group after 6 weeks began to make 20% fewer search appeals, which ended on the first page. That is, Google began to receive 20% fewer requests that ended on the first page.

Another possible reason could be that the respondents fell victim to the “paradox of choice”: when there are so many options, it is difficult to make a choice.

Perhaps, but the researchers also found that there was another “variable” in this experiment that they could not influence - this is the page loading speed.

Analyzing the site logs, they found that loading the page with the 30th search results took almost twice as long.

“Collect” the page with the 30th search results: select them from the database, take into account the preferences of this user, format it all on the screen and show, it took more than 2 times longer. To be precise: 0.4 seconds for a page with 10 results, and 0.9 seconds for a page with 30 results.

And this is still very fast, because as Marissa Mayer said, a typical search query causes a call to 700-1000 different servers from within Google.

Therefore, the company concluded that perhaps the users were right in their potential desires. They really wanted more search results. But what they didn’t talk about, and perhaps they couldn’t know, is that “more results” means “the page loads longer”. And, of course, they could not predict in advance their own reactions to this.

It seems that if any user asks "how important is the 0.5 second delay for you to download the site," not many people will answer "this is very critical, I will think about the service worse and gradually will use it less." But real tests show exactly that.

The results shown are important from three points of view. First, it shows a general error in UX and experiments with usability: when an experiment that is incorrectly performed causes an incorrect interpretation of the results.

Secondly, we are again reminded that what is not always what the users are talking about is actually what they want. We recall the words of Henry Ford “if I asked customers what they need, their answer would be a faster horse” and Steve Jobs “no focus group would have invented an ipad”.

Of course, feedback from users is important, but before implementing any actions on it, you must first test how the "words" of users are combined with their "deeds."

How much does the way they behave with the product really relate to what they say? And in the end - how their changed behavior will affect the company's revenue.

After all, a great product that reduces the company's revenue is a very strange business decision.