Find for spy: AAB-testing as the best option for split-test
What gives the addition of one more letter “A” to the good old AB test and why this is also not enough to assess the changes.
An AAB test or an ABA test is when traffic is divided into three parts: the first and second are sent to the site without changes, and the third with changes.
1. Insufficient traffic / operations
The conclusion is done early, high error.
')
2. Incorrect rotation
A system or script that is engaged in rotation does not give an arbitrary and even distribution between the test blocks.
3. Incomplete Cross Browser
In some of the browsers, the change is not working correctly. A group of users of this browser may have features. Excluding them from the test distorts its result.
4. Influence of devices
On different devices, the changes may look different. For example, on large screens, small details are not so noticeable.
5. The effect of novelty
Users noticed a new element on the site or its change, they pay more attention to it and it works better at first.
6. The effect of addiction
Users are accustomed to some element and do not notice its change, working with it as with the old.
7. Misses in the buying cycle
During the evaluation of the conversion, the test period may not coincide with the cycle in which the purchase takes place. Other factors may work earlier, and at the time of the test, users will already be prepared, the test will show an incorrect result. Also, the consumer cycle may have a long-lasting or delayed effect and it will not fall into the test results.
8. Other changes
It is very difficult to achieve a clean test, while other changes may occur on the site. At first glance it may seem that they are not related, but this will indirectly affect the test.
9. Seasonal factors, sales and changes in commodity balances and others.
In my experience it is:
Each project has its own figures and just the AAB test will help determine them.
Much has been written about this, I see no reason to repeat. Perhaps it makes sense to give links to services with which you can do tests including multivariate:
Experiments in Google Analitycs - a great option for a beginner.
Changeagain.me , Convert.com , Vwo.com, Optimizely.com - powerful, paid services with integration with an account in Google Analytics.
A / Bingo - server frameworks for Ruby.
In the Google Adsense advertising system, the function of experiments is also there.
I also note the online calculator of criteria for the significance of the split test
and a graphing service based on its results.
To estimate the correct time and the required amount of data, AA tests are sometimes used when the same is put on the test, without changes. This approach has the right to life, but it takes time. In addition, from the time of the AA to the AB test, something may change.
Summing up, we can say that AAB tests require more resources (traffic, time, money), but they give a more accurate and confident result. To improve it, they also use the AABB system. In any case, for a new project or a type of change, it is worth doing the AAB test once or twice, then you can switch to classic AB.
All good tests!
See also: How to get maximum revenue from advertising systems on your website .
An AAB test or an ABA test is when traffic is divided into three parts: the first and second are sent to the site without changes, and the third with changes.
What does this give?
- Higher test accuracy. We believe that B gives a relatively reliable result only when A and A become equal.
- There are sharp fluctuations in the values of A and A, it is easier to understand which of the factors that distort the test at this point affected.
- We get the time and amount of data needed for future tests.
Factors distorting test results
1. Insufficient traffic / operations
The conclusion is done early, high error.
')
2. Incorrect rotation
A system or script that is engaged in rotation does not give an arbitrary and even distribution between the test blocks.
3. Incomplete Cross Browser
In some of the browsers, the change is not working correctly. A group of users of this browser may have features. Excluding them from the test distorts its result.
4. Influence of devices
On different devices, the changes may look different. For example, on large screens, small details are not so noticeable.
5. The effect of novelty
Users noticed a new element on the site or its change, they pay more attention to it and it works better at first.
6. The effect of addiction
Users are accustomed to some element and do not notice its change, working with it as with the old.
7. Misses in the buying cycle
During the evaluation of the conversion, the test period may not coincide with the cycle in which the purchase takes place. Other factors may work earlier, and at the time of the test, users will already be prepared, the test will show an incorrect result. Also, the consumer cycle may have a long-lasting or delayed effect and it will not fall into the test results.
8. Other changes
It is very difficult to achieve a clean test, while other changes may occur on the site. At first glance it may seem that they are not related, but this will indirectly affect the test.
9. Seasonal factors, sales and changes in commodity balances and others.
What can be done to reduce these effects?
- conduct a split test only on new users or on new and old separately.
- use multiple systems to control.
- select a separate source of traffic for the test.
- do AAB tests.
Test Minimum Values
In my experience it is:
for advertising campaigns 500.000, a period of 2 weeks.
For website conversion: 500 transactions, 2 months.
to evaluate small changes on the site: 3 thousand sessions, a week.
Each project has its own figures and just the AAB test will help determine them.
And what if there is no such amount on the site?
- To accept that the split tests while you do not work.
- Leave the tests working, let them collect statistics.
- Use other ways to test hepatosis (surveys, competitor research, user behavior assessment).
- Work on minimizing distortion in future tests. Sometime their time will come, and you are ready.
Classic AB testing
Much has been written about this, I see no reason to repeat. Perhaps it makes sense to give links to services with which you can do tests including multivariate:
Experiments in Google Analitycs - a great option for a beginner.
Changeagain.me , Convert.com , Vwo.com, Optimizely.com - powerful, paid services with integration with an account in Google Analytics.
A / Bingo - server frameworks for Ruby.
In the Google Adsense advertising system, the function of experiments is also there.
I also note the online calculator of criteria for the significance of the split test
and a graphing service based on its results.
AA testing
To estimate the correct time and the required amount of data, AA tests are sometimes used when the same is put on the test, without changes. This approach has the right to life, but it takes time. In addition, from the time of the AA to the AB test, something may change.
Pros and cons of AAB
Summing up, we can say that AAB tests require more resources (traffic, time, money), but they give a more accurate and confident result. To improve it, they also use the AABB system. In any case, for a new project or a type of change, it is worth doing the AAB test once or twice, then you can switch to classic AB.
All good tests!
See also: How to get maximum revenue from advertising systems on your website .
Source: https://habr.com/ru/post/323702/
All Articles