The biggest mistake in A / B testing
Want to have a high conversion website? Everyone wants. But many make this mistake. An error that misleads them and leads to a sad result.
The guys from Popcorn Metrics - made this mistake in A / B testing. And to help others not to be trapped, they wrote an article about it. And we quickly translated it to warn you of this error!
Let's imagine the situation
')
You conduct A / B testing site . You have two versions - original and test. As a result of testing, you get the following results: conversion to registration on the original page - 2.3%, in the test version - 3.2%. Which of these pages won the experiment?
Of course, the test page is more efficient. With this we can not agree at this stage. Why? Because the numbers do not lie. 3.2% more than 2.3%.
And here the fun begins. When people do A / B testing, they focus on only one stage of the sales funnel. In our example, for conversion to registration.
It would seem, what is the problem? Everywhere you are taught to choose a specific goal when creating an A / B experiment and to evaluate the result according to it. But almost never says that you need to look further, deeper.
After all, there are other stages of the sales funnel. In your case, more users registered in the test version. That's cool. But how did they proceed? How did they convert to paid users? This question is much more important.
This is the problem. Practically no one assesses the impact of an A / B experiment on the entire sales funnel, starting with the first stage that you tried to improve with this test, and ending with the last stage - buying, for example.
Now let's continue with this experiment. Despite the fact that the test version of the page had more conversions into registration, the number of sales in the original version turned out to be more. Now is not everything so obvious? You can already disagree with the fact that the test version won the A / B experiment.
What is more important for you: increase the number of conversions in the first stage of the sales funnel or improve the entire sales funnel so that more users reach the final goal?
The article in the topic - the 10 most common errors in A / B testing
Attention! If during A / B testing your goal is to increase conversion only at one specific stage, for example at the Registration stage, regardless of how this affects the entire sales funnel and your income, then you can stop reading.
If you are watching “next” and want to do the right, effective A / B tests, then stay with us and we will tell and show what mistake the guys from Popcorn Metrics made. Perhaps you are doing it right now and not even guessing about it.
Let's look at the sales funnel project Popcorn metrics. There are 5 stages in it, starting with registration in the service (Register) and ending with the placement of the code (Publish), and you can see what the conversion rate is from one stage to another. The main task in this case is to bring the potential client to the last stage, when he places the tracking code on his website.
Article in topic - 10 successful cases A / B testing
When a potential client got to the main page of Popcorn Metrics, he was asked to enter a link to his site. After inserting the link and clicking on the "Try Now" button, the visual editor (Playmode) opens. In this visual editor you can see how the service works even without registration. This is done in order to show the benefits of the service as quickly as possible, while not forcing a potential client to register. First, the value, and then the obligation - registration for the trial version of the product.
The effectiveness of this function has been tested in previous experiments. It was decided to continue the experiments and test options to add the ability to view a 2-minute video.
In the 1st version, a video was added directly to the main page. In the 2nd - the text link to the video.
"
An experiment was launched using Google Analytics. “Registration” was chosen as the main goal - the first stage of the sales funnel.
After the experiment, it turned out that in the 1st test version with the inserted video (Variaton 1 - Playmode & Video Image) the conversion into registration is 3.20%. And it was the best indicator among all three versions.
Thus, the winner in this experiment is the first test version. His conversion was 40% more than in the other two cases. And this is where the biggest mistake is hidden.
When conducting experiments in Google Analytics, you have the opportunity to look at the results of the experiment in relation to all established goals. Those. even if you chose “Registration” as the main goal - the first stage of the sales funnel, you can also track the effect of the experiment on other goals.
Article in the topic - How to run a split test: 13 required steps
The guys from Popcorn Metrics just added another goal for analyzing the experiment - Publish code. If you suddenly forgot, then this is the ultimate goal in the sale funnel. As a result, it turned out that our pseudo-winner (Variation 1) has the lowest conversion rate for this goal. And the original version of the page is the highest.
Thus, in the test version with the video on the page, people most often registered, but at the same time almost 2.5 times less often reached the end of the sales funnel.
When a potential client begins his acquaintance with the service through a visual editor, he alone forms a certain user experience. When through video, then another experience. The subsequent actions depend on it.
With the help of analytics another interesting conclusion was also made. Those who first watched the video very rarely tested the visual editor (in 9.62% of cases).
Despite the fact that the video was very revealing and the users watched it, it did not fascinate them like a visual editor, as a result of which they were not converted to the next stages of the sales funnel.
It is very dangerous to conduct an A / B experiment and analyze only one goal, regardless of the entire sales funnel. You can make the wrong conclusion, which will seem right at first glance. And if you look deeper, then everything will be different.
Just imagine. You conducted an experiment, made this error and continued testing other elements of the site, depending on the results of this test. It turns out that all further experiments will lead you even more into error and then it will be difficult to understand what went wrong, why the conversion at the first stage increases, and incomes fall.
Now you know about the biggest error in A / B testing, which many still admit and do not even know about it. Be smart, test correctly!
Prepared by the blog Boosta.ru and the project Changeagain.me
Read also our other popular articles:
1. They will help you find all the “killers” of the conversion - 10 reports in Google Analytics (5000+ views)
2. 100 ideas for A / B testing. Part One (2000+ views)
3. 100 ideas for A / B testing: Part Two (1000+ views)
4. How to find an idea for an A / B test: heat maps and polls (1000+ views)
The guys from Popcorn Metrics - made this mistake in A / B testing. And to help others not to be trapped, they wrote an article about it. And we quickly translated it to warn you of this error!
Let's imagine the situation
')
You conduct A / B testing site . You have two versions - original and test. As a result of testing, you get the following results: conversion to registration on the original page - 2.3%, in the test version - 3.2%. Which of these pages won the experiment?
Of course, the test page is more efficient. With this we can not agree at this stage. Why? Because the numbers do not lie. 3.2% more than 2.3%.
And here the fun begins. When people do A / B testing, they focus on only one stage of the sales funnel. In our example, for conversion to registration.
It would seem, what is the problem? Everywhere you are taught to choose a specific goal when creating an A / B experiment and to evaluate the result according to it. But almost never says that you need to look further, deeper.
After all, there are other stages of the sales funnel. In your case, more users registered in the test version. That's cool. But how did they proceed? How did they convert to paid users? This question is much more important.
This is the problem. Practically no one assesses the impact of an A / B experiment on the entire sales funnel, starting with the first stage that you tried to improve with this test, and ending with the last stage - buying, for example.
Now let's continue with this experiment. Despite the fact that the test version of the page had more conversions into registration, the number of sales in the original version turned out to be more. Now is not everything so obvious? You can already disagree with the fact that the test version won the A / B experiment.
What is more important for you: increase the number of conversions in the first stage of the sales funnel or improve the entire sales funnel so that more users reach the final goal?
The article in the topic - the 10 most common errors in A / B testing
Attention! If during A / B testing your goal is to increase conversion only at one specific stage, for example at the Registration stage, regardless of how this affects the entire sales funnel and your income, then you can stop reading.
If you are watching “next” and want to do the right, effective A / B tests, then stay with us and we will tell and show what mistake the guys from Popcorn Metrics made. Perhaps you are doing it right now and not even guessing about it.
Case of Popcornmetrics
Let's look at the sales funnel project Popcorn metrics. There are 5 stages in it, starting with registration in the service (Register) and ending with the placement of the code (Publish), and you can see what the conversion rate is from one stage to another. The main task in this case is to bring the potential client to the last stage, when he places the tracking code on his website.
Article in topic - 10 successful cases A / B testing
When a potential client got to the main page of Popcorn Metrics, he was asked to enter a link to his site. After inserting the link and clicking on the "Try Now" button, the visual editor (Playmode) opens. In this visual editor you can see how the service works even without registration. This is done in order to show the benefits of the service as quickly as possible, while not forcing a potential client to register. First, the value, and then the obligation - registration for the trial version of the product.
The effectiveness of this function has been tested in previous experiments. It was decided to continue the experiments and test options to add the ability to view a 2-minute video.
In the 1st version, a video was added directly to the main page. In the 2nd - the text link to the video.
"An experiment was launched using Google Analytics. “Registration” was chosen as the main goal - the first stage of the sales funnel.
After the experiment, it turned out that in the 1st test version with the inserted video (Variaton 1 - Playmode & Video Image) the conversion into registration is 3.20%. And it was the best indicator among all three versions.
Thus, the winner in this experiment is the first test version. His conversion was 40% more than in the other two cases. And this is where the biggest mistake is hidden.
See the results of the experiment at all stages of the funnel.
When conducting experiments in Google Analytics, you have the opportunity to look at the results of the experiment in relation to all established goals. Those. even if you chose “Registration” as the main goal - the first stage of the sales funnel, you can also track the effect of the experiment on other goals.
Article in the topic - How to run a split test: 13 required steps
The guys from Popcorn Metrics just added another goal for analyzing the experiment - Publish code. If you suddenly forgot, then this is the ultimate goal in the sale funnel. As a result, it turned out that our pseudo-winner (Variation 1) has the lowest conversion rate for this goal. And the original version of the page is the highest.
Thus, in the test version with the video on the page, people most often registered, but at the same time almost 2.5 times less often reached the end of the sales funnel.
Why did it happen?
When a potential client begins his acquaintance with the service through a visual editor, he alone forms a certain user experience. When through video, then another experience. The subsequent actions depend on it.
With the help of analytics another interesting conclusion was also made. Those who first watched the video very rarely tested the visual editor (in 9.62% of cases).
Despite the fact that the video was very revealing and the users watched it, it did not fascinate them like a visual editor, as a result of which they were not converted to the next stages of the sales funnel.
What is the lesson to learn from this case?
It is very dangerous to conduct an A / B experiment and analyze only one goal, regardless of the entire sales funnel. You can make the wrong conclusion, which will seem right at first glance. And if you look deeper, then everything will be different.
Just imagine. You conducted an experiment, made this error and continued testing other elements of the site, depending on the results of this test. It turns out that all further experiments will lead you even more into error and then it will be difficult to understand what went wrong, why the conversion at the first stage increases, and incomes fall.
Now you know about the biggest error in A / B testing, which many still admit and do not even know about it. Be smart, test correctly!
Prepared by the blog Boosta.ru and the project Changeagain.me
Read also our other popular articles:
1. They will help you find all the “killers” of the conversion - 10 reports in Google Analytics (5000+ views)
2. 100 ideas for A / B testing. Part One (2000+ views)
3. 100 ideas for A / B testing: Part Two (1000+ views)
4. How to find an idea for an A / B test: heat maps and polls (1000+ views)
Source: https://habr.com/ru/post/295368/
All Articles