Algorithms and Conversion: How we searched for the best moment to call back
A widget for ordering a callback can be very useful if it appears at the right moment. But how to calculate when a visitor “ripens” before talking on the phone?
From the very start of the project, it was obvious that it was necessary to constantly work on improving the efficiency of our widget - this is the only way to bypass competitors (which, as we know, quite a lot already).
')
The conversion is greatly influenced by choosing the right moment. If the widget “pops up” at the moment when the user has a question, then he is more likely to agree to the call and will not experience negative emotions. This moment is the real “Holy Grail” of all callback widgets, but finding it is not so easy.
Our path has already taken 4 stages, and we hope that we have successfully completed.
At the time of the start of testing, the number of unique visitors to the sites on which the Cashmyvisit widget stood was about 20 thousand per month.
We decided to check how the conversion time delay affects the appearance of the widget on the site - 30, 40, 50 seconds. All connected sites were in the initial sample, the number of calls per 100 visitors was estimated.
It turned out that too early appearance of the widget leads to an increase in the number of failures. In addition, we noticed that various external factors, which could not be influenced in any way, exert a strong influence on the results - as a result, the figures, from week to week, were very different. It's not that.
When the first stage was in full swing, the team of our project came to the IIDF conference in St. Petersburg. There, we listened to a speech by Grigoriy Bakunov, director of technology distribution at Yandex, who very aptly mentioned A / B testing. Later, we found a video on YouTube about how he used the machine learning and such tests to promote the CRM system in the USA - the mechanism seemed suitable for our tasks.
We began to introduce A / B testing into the development process. In a short time, five scenarios of the appearance of the widget were launched, starting with a timeout of 37 seconds in increments of 8 seconds.
Unfortunately, hopes were not justified. After a month of tests, it became clear that:
Do all users need to show the widget? At the next stage, we decided to identify the most involved site visitors, and offer a call back only to them. To do this, it was necessary to develop criteria for "interest". We attributed to them:
In addition, additional “engagement points” were awarded in two cases:
The combination of such behavioral factors should signal that the appearance of the widget will be timely for the visitor.
Just the next day after the introduction of the described mathematical model into the prototype of the widget, one of the competitors “rolled out” an update that includes everything with which we planned to become market leaders!
It was a blow to our vanity, but we brought it to the end and released our own update. The collected data showed that the product still does not work as we would like.
Then we attracted to the work of professional mathematicians who pointed out the main error. We and the competitors, the time of the user on the site was as the most important indicator in the entire system. In 9 different parameters, the degree of correlation with the “time” parameter was more than 80% - as a result, the system did not control how many people the widget would be shown and did not correlate its behavior with the theme of a particular site.
It became clear - this system is far from the Grail.
It is the support vector method - we were recommended to use mathematics.
We decided to try it and after a couple of hours the development department, together with mathematicians, wrote out formulas of paper with formulas.
The resulting system includes several parameters in 230 sections (pages of the site, sources of visits, time on the site, time of day, day of the week, etc.). During the analysis, natural logarithms are calculated, the values are summed, this sum is compared with the unit. If the result is equal to or greater than one, the widget is shown to the visitor. Values are recalculated every second.
It took several months to refine the system; in April, we launched it in test mode on a small sample of client sites. Already the first measurements showed an increase in conversion, on average, by 20-25%.
It took us more than half a year to complete all the searches, tests and mistakes described above. However, the resulting result pleases both us and our customers (here we talked about how the use of the widget helped increase the loyalty of the auto dealers' customers). We are not going to stop there and continue to refine the system so that the conversion is even higher.
After all, continuous development is the only way to win in a competitive market like ours.
From the very start of the project, it was obvious that it was necessary to constantly work on improving the efficiency of our widget - this is the only way to bypass competitors (which, as we know, quite a lot already).
')
The conversion is greatly influenced by choosing the right moment. If the widget “pops up” at the moment when the user has a question, then he is more likely to agree to the call and will not experience negative emotions. This moment is the real “Holy Grail” of all callback widgets, but finding it is not so easy.
Our path has already taken 4 stages, and we hope that we have successfully completed.
Stage One: Different Widget Appearances
At the time of the start of testing, the number of unique visitors to the sites on which the Cashmyvisit widget stood was about 20 thousand per month.
We decided to check how the conversion time delay affects the appearance of the widget on the site - 30, 40, 50 seconds. All connected sites were in the initial sample, the number of calls per 100 visitors was estimated.
It turned out that too early appearance of the widget leads to an increase in the number of failures. In addition, we noticed that various external factors, which could not be influenced in any way, exert a strong influence on the results - as a result, the figures, from week to week, were very different. It's not that.
Stage Two: A / B Tests
When the first stage was in full swing, the team of our project came to the IIDF conference in St. Petersburg. There, we listened to a speech by Grigoriy Bakunov, director of technology distribution at Yandex, who very aptly mentioned A / B testing. Later, we found a video on YouTube about how he used the machine learning and such tests to promote the CRM system in the USA - the mechanism seemed suitable for our tasks.
We began to introduce A / B testing into the development process. In a short time, five scenarios of the appearance of the widget were launched, starting with a timeout of 37 seconds in increments of 8 seconds.
Unfortunately, hopes were not justified. After a month of tests, it became clear that:
- He does not give food for real conclusions.
- Data for the entire system is generally irrelevant for one specific site.
- Conclusions should be made not upon visiting the site, but upon the activation of the widget.
Stage Three: Criteria for Determining Interest
Do all users need to show the widget? At the next stage, we decided to identify the most involved site visitors, and offer a call back only to them. To do this, it was necessary to develop criteria for "interest". We attributed to them:
- The activity time on the site is more than 30 seconds (based on mouse tracking).
- User activity is longer than the average activity of other visitors.
- The depth of the view exceeds the average figures.
- The source of the transition was a contextual advertisement.
In addition, additional “engagement points” were awarded in two cases:
- For every 15 seconds, beyond the average time spent on the site.
- For the implementation of 100% page scrolling.
The combination of such behavioral factors should signal that the appearance of the widget will be timely for the visitor.
Just the next day after the introduction of the described mathematical model into the prototype of the widget, one of the competitors “rolled out” an update that includes everything with which we planned to become market leaders!
It was a blow to our vanity, but we brought it to the end and released our own update. The collected data showed that the product still does not work as we would like.
Then we attracted to the work of professional mathematicians who pointed out the main error. We and the competitors, the time of the user on the site was as the most important indicator in the entire system. In 9 different parameters, the degree of correlation with the “time” parameter was more than 80% - as a result, the system did not control how many people the widget would be shown and did not correlate its behavior with the theme of a particular site.
It became clear - this system is far from the Grail.
Fourth Stage: Support Vectors
It is the support vector method - we were recommended to use mathematics.
We decided to try it and after a couple of hours the development department, together with mathematicians, wrote out formulas of paper with formulas.
The resulting system includes several parameters in 230 sections (pages of the site, sources of visits, time on the site, time of day, day of the week, etc.). During the analysis, natural logarithms are calculated, the values are summed, this sum is compared with the unit. If the result is equal to or greater than one, the widget is shown to the visitor. Values are recalculated every second.
It took several months to refine the system; in April, we launched it in test mode on a small sample of client sites. Already the first measurements showed an increase in conversion, on average, by 20-25%.
Rejoice early
It took us more than half a year to complete all the searches, tests and mistakes described above. However, the resulting result pleases both us and our customers (here we talked about how the use of the widget helped increase the loyalty of the auto dealers' customers). We are not going to stop there and continue to refine the system so that the conversion is even higher.
After all, continuous development is the only way to win in a competitive market like ours.
Source: https://habr.com/ru/post/290646/
All Articles