Web Analytics: Not all numbers are equally useful.
We are asked all the time: what is the error of data collection in Google Analytics? Which counter is better to trust? Is it possible to get rid of all discrepancies and get exact numbers of attendance?
When we run an advertising campaign, we pay for each visitor. These web analysts should tell us which ads are wasting their budget in vain, which keywords are more important to us. We need to know the result as soon as possible! When will it be ready?
Common sense in solving this problem is not an assistant. The fact is that in the analysis of statistical indicators, we are faced with random processes. Our brain tries to imagine a uniform process: if the conversion on the site is 10%, it seems to us that every tenth will be converted:
')
Even if we understand that reality looks different, it is difficult to get rid of the illusion: “wait for a few conversions and get accurate data.” In fact, this process is much more chaotic, for example:
This means that with a limited number of visits, even with accurate data on the number of visitors, views, conversions, we cannot accurately predict future sales. There is no such amount of data that provides solid confidence in the calculated indicators: we can only roughly estimate the “real” percentage of conversion.
There is also good news: we can estimate the accuracy with which we calculated this coefficient, and realize whether this accuracy is enough for us, or if we need to wait for more data. I will not delve into the mathematical calculations, telling the theoretical part. I will give only two simple formulas by which it is easy to calculate the numbers you need.
The main concept of mathematical statistics that every analyst should learn is the confidence interval. This is the range in which the true value of the value we need may lie: for example, the conversion rate. The value of the conversion calculated in the usual way (for example, by the Google Analytics system) lies somewhere within this interval; the true value of the conversion (which we learn only on endless traffic) is also most likely inside it.
As the data sample is increased, the bell narrows, and the probability that the true value is closer to the measured one increases. We choose what reliability is considered sufficient. Different confidence intervals can be constructed for the same curve. The graph shows two options (red and green).
Let's get down to business. Let's calculate the confidence interval for the conversion of one of the advertisements - “Buy an elephant at a discount!”. For example, in Google Analytics we see 143 visits and a conversion of 2.1% for this ad. We want to compare this ad with another (“Elephant with delivery in Moscow”) - he has 184 visits and 2.7% conversion - and choose the best. Many marketers have already concluded that the second ad is better, but we need to check it mathematically.
Before starting the analysis, we make an important decision: what reliability will be sufficient. The most convenient way is to use standard values ​​that simplify calculations: these are 68%, 95% and 99.7% (they are called “one sigma”, “two sigma” and “three sigma”, respectively). I would like to count with maximum certainty (three sigmas), but this expands the interval and often makes it completely uninformative.
The formula that will help us is as follows:
,
where R is the simple conversion ratio calculated by simple division, N is the number of visits, and α is the number of sigmas, i.e. measurement accuracy.
With an accuracy of 68%, the conversion of the first ad ranges from 0.8% to 3.3% ; second ad conversion from 1.7% to 3.8% .
Discouraging, is not it? It is impossible to compare the efficiency: the intervals overlap. Both ads may be leaders. If we want to build intervals for a probability of 95%, they will be two times wider. We need to wait for the data, and the trouble is that we don’t even know when these intervals will “part” and the leader will come to light: it depends on the difference between the true conversion rates. Usually, for definiteness, you need about a hundred conversions for each ad (word, campaign or other segment, the conversion of which we consider). Please note: our formula counts well only when the conversion does not exceed 30%.
And what if our ad did not give any conversion? Toad begins to choke us: let's quickly turn off advertising and save the budget!
Zero conversions cannot be considered using the previous formula: it will say that 0 is the exact result. This is not true. If we have 0 conversions and N visits, then the confidence interval is considered simple :
If we have a third announcement (“Elephants as selection”), which resulted in 93 visitors without a single conversion, we can only say with certainty that its conversion is below 1.2%. This ad is worse than “Elephant with delivery”, but it’s too early to talk about “Elephant at a discount”. Most people in this place are surprised, and some do not even believe it: is it really that shaky? However, the experience of carrying out large campaigns only confirms the random nature of the conversions and the fact that at first the “null” ad on the second hundred clicks can begin to show good results.
Why popular analytics do not consider these values ​​for us? I can only guess. This information will significantly complicate the reports, and most importantly, will expose the unpleasant truth: a noticeable number of smart and accurate figures in these reports are actually useless and even dangerous (if you rely on common sense). Forgetting about counting errors, you can stop an effective advertising campaign, finding it unsuccessful, choose a bad ad option, distribute the advertising budget incorrectly, or make a mistake when developing an SEO strategy.
Confidence intervals are considered excellent by Google Website Optimizer, a tool for conducting and analyzing test results on sites. Optimizer draws conclusions only when it is convinced that the intervals have "diverged" and one of the variants of the page statistically reliably surpasses the rest of the samples in conversion.
Many tools take this data into account “under the hood”: for example, Yandex.Direct makes conclusions about the announcement only when it collects enough data and is confident in the result. Of course, it would be interesting for us to see them in traffic reports as well, so as not to err in the estimates “by eye”. You can write a script for Excel that automatically calculates confidence intervals, but this requires constant data upload. You can add this functionality to Metrics or Analytics using the APIs of these systems and external scripts.
Do we really have to wait for hundreds of conversions for each ad in order to evaluate its performance? This would be an ideal solution, but usually impossible: there will not be enough advertising budget. Try not to make premature fatal decisions (to remove or modify ads that have not proved their inconsistency) before obtaining statistically significant results; while nothing prevents, for example, temporarily reduce the cost of a click on them.
It is useful to add an account of intermediate goals : for example, placing goods in the cart is a sign of an interested buyer, but there will be more such conversions than confirmed purchases, so you will see a statistically significant result earlier. And at the same time, estimate how many people leave the site due to the uncomfortable process of checkout!
Do not be disappointed in the numbers, but beware: then you will be able to see a guide to action where others find only a dirty lie.
We always answer: the error is usually about 10%, there is no obvious leader in accuracy, it is impossible to remove all errors - this is the way the technology works.Almost no one understands that inaccurate data collection is not the only error affecting the analysis result. Even ideally collected data will not allow us to accurately calculate the necessary indicators on the site (first of all, the percentage of conversion). Collected data may not be enough! Everyone understands this: if only 15 visitors came to the site and none of them filled out the loan application form, it’s too early to talk about conversion. So tells us common sense; but at what point can we say that there is enough data? Should I wait another 100 visits? 200? 500?
When we run an advertising campaign, we pay for each visitor. These web analysts should tell us which ads are wasting their budget in vain, which keywords are more important to us. We need to know the result as soon as possible! When will it be ready?
Common sense in solving this problem is not an assistant. The fact is that in the analysis of statistical indicators, we are faced with random processes. Our brain tries to imagine a uniform process: if the conversion on the site is 10%, it seems to us that every tenth will be converted:
')
Even if we understand that reality looks different, it is difficult to get rid of the illusion: “wait for a few conversions and get accurate data.” In fact, this process is much more chaotic, for example:
This means that with a limited number of visits, even with accurate data on the number of visitors, views, conversions, we cannot accurately predict future sales. There is no such amount of data that provides solid confidence in the calculated indicators: we can only roughly estimate the “real” percentage of conversion.
There is also good news: we can estimate the accuracy with which we calculated this coefficient, and realize whether this accuracy is enough for us, or if we need to wait for more data. I will not delve into the mathematical calculations, telling the theoretical part. I will give only two simple formulas by which it is easy to calculate the numbers you need.
The main concept of mathematical statistics that every analyst should learn is the confidence interval. This is the range in which the true value of the value we need may lie: for example, the conversion rate. The value of the conversion calculated in the usual way (for example, by the Google Analytics system) lies somewhere within this interval; the true value of the conversion (which we learn only on endless traffic) is also most likely inside it.
Probably?!Yes, we can only say with absolute accuracy that the conversion is non-negative (and less than one hundred percent). Mathematical formulas will help us build a confidence interval in which the actual conversion will be very likely. The probability distribution for our estimate is as follows:
As the data sample is increased, the bell narrows, and the probability that the true value is closer to the measured one increases. We choose what reliability is considered sufficient. Different confidence intervals can be constructed for the same curve. The graph shows two options (red and green).
Let's get down to business. Let's calculate the confidence interval for the conversion of one of the advertisements - “Buy an elephant at a discount!”. For example, in Google Analytics we see 143 visits and a conversion of 2.1% for this ad. We want to compare this ad with another (“Elephant with delivery in Moscow”) - he has 184 visits and 2.7% conversion - and choose the best. Many marketers have already concluded that the second ad is better, but we need to check it mathematically.
Before starting the analysis, we make an important decision: what reliability will be sufficient. The most convenient way is to use standard values ​​that simplify calculations: these are 68%, 95% and 99.7% (they are called “one sigma”, “two sigma” and “three sigma”, respectively). I would like to count with maximum certainty (three sigmas), but this expands the interval and often makes it completely uninformative.
The formula that will help us is as follows:
,where R is the simple conversion ratio calculated by simple division, N is the number of visits, and α is the number of sigmas, i.e. measurement accuracy.
With an accuracy of 68%, the conversion of the first ad ranges from 0.8% to 3.3% ; second ad conversion from 1.7% to 3.8% .
Discouraging, is not it? It is impossible to compare the efficiency: the intervals overlap. Both ads may be leaders. If we want to build intervals for a probability of 95%, they will be two times wider. We need to wait for the data, and the trouble is that we don’t even know when these intervals will “part” and the leader will come to light: it depends on the difference between the true conversion rates. Usually, for definiteness, you need about a hundred conversions for each ad (word, campaign or other segment, the conversion of which we consider). Please note: our formula counts well only when the conversion does not exceed 30%.
And what if our ad did not give any conversion? Toad begins to choke us: let's quickly turn off advertising and save the budget!
Zero conversions cannot be considered using the previous formula: it will say that 0 is the exact result. This is not true. If we have 0 conversions and N visits, then the confidence interval is considered simple :
- For one sigma (68% confidence): interval from 0 to 1.15 / N
- For two sigma (95% confidence): interval from 0 to 3 / N
- For three sigma (99.7% confidence): range from 0 to 6 / N.
If we have a third announcement (“Elephants as selection”), which resulted in 93 visitors without a single conversion, we can only say with certainty that its conversion is below 1.2%. This ad is worse than “Elephant with delivery”, but it’s too early to talk about “Elephant at a discount”. Most people in this place are surprised, and some do not even believe it: is it really that shaky? However, the experience of carrying out large campaigns only confirms the random nature of the conversions and the fact that at first the “null” ad on the second hundred clicks can begin to show good results.
Why popular analytics do not consider these values ​​for us? I can only guess. This information will significantly complicate the reports, and most importantly, will expose the unpleasant truth: a noticeable number of smart and accurate figures in these reports are actually useless and even dangerous (if you rely on common sense). Forgetting about counting errors, you can stop an effective advertising campaign, finding it unsuccessful, choose a bad ad option, distribute the advertising budget incorrectly, or make a mistake when developing an SEO strategy.
Confidence intervals are considered excellent by Google Website Optimizer, a tool for conducting and analyzing test results on sites. Optimizer draws conclusions only when it is convinced that the intervals have "diverged" and one of the variants of the page statistically reliably surpasses the rest of the samples in conversion.
Many tools take this data into account “under the hood”: for example, Yandex.Direct makes conclusions about the announcement only when it collects enough data and is confident in the result. Of course, it would be interesting for us to see them in traffic reports as well, so as not to err in the estimates “by eye”. You can write a script for Excel that automatically calculates confidence intervals, but this requires constant data upload. You can add this functionality to Metrics or Analytics using the APIs of these systems and external scripts.
Do we really have to wait for hundreds of conversions for each ad in order to evaluate its performance? This would be an ideal solution, but usually impossible: there will not be enough advertising budget. Try not to make premature fatal decisions (to remove or modify ads that have not proved their inconsistency) before obtaining statistically significant results; while nothing prevents, for example, temporarily reduce the cost of a click on them.
It is useful to add an account of intermediate goals : for example, placing goods in the cart is a sign of an interested buyer, but there will be more such conversions than confirmed purchases, so you will see a statistically significant result earlier. And at the same time, estimate how many people leave the site due to the uncomfortable process of checkout!
Do not be disappointed in the numbers, but beware: then you will be able to see a guide to action where others find only a dirty lie.
Source: https://habr.com/ru/post/125446/
All Articles