Cases successful (and not so) experiments "Yandex. Navigator"
The Yandex.Navigator product team shared what experiments are being conducted, how work is organized inside, how the product is monetized, and what approaches it uses in predictive analytics.
Epic talks is the Epic Growth project, where we communicate with teams about proven processes, testing of product hypotheses, analytics and much more. The project was filmed with the support of the AppMetrica analytical platform.
Lyosha Gozhev, Senior Food Analyst and Misha Vysokovsky, Head of Yandex.Navigator
')
How is the product team Yandex. Navigator?
Inside the Navigator, we have several product teams, each of which has its own focus. Focus sets the direction for product development. For example, one team takes care of the route, the other focuses on the search script. Unlike a formed product team, the focus may change over time. Usually each team has a product manager, designer and developer.
We now have two analytics, so they work for several product teams. Mostly analyst tasks are divided into two types.
The first type is adhoc-tasks that need to be calculated quickly, or tasks requiring verification of code writing. The second type is research tasks in which you need to dive with the whole team. In such tasks there is no separation between the performer and the customer. The whole process of research tasks is very similar to traversing a growing tree: we generate hypotheses and then test them. During their refutation or confirmation new hypotheses arise.
Give an example of successful experiments?
We try to guess where the person will go and offer him the “destination suggest”. For example, the user opens the navigator, and the application offers to go on the most frequent routes. Now we have about 13% of the routes being built this way. This indicator is constantly growing. We are engaged in improving the predictive model, we look at how it affects and how much more convenient it becomes for the user.
Another example of a successful experiment is interface redesign using conventional electrical tape. Since we have a specific product, testing the Navigator while sitting at the table is a waste of time. At some point, we looked at the interface of J. Navigator and realized that there was a lot of duplicate information in it.
We thought: “Listen, the third screen on top is occupied by a huge black panel. Maybe we should throw her out and everything will be fine without her? ” The easiest option is to take the usual electrical tape and glue the top plate.
The first feedback, which appeared - is "I understand where to go, but it is not clear when I come." We took the scissors and cut them a little so that the time of arrival could be seen. We were surprised by the positive feedback. People quietly reached the place without information, which was sealed with black electrical tape, despite the fact that they were driving along an unfamiliar route.
How many hypotheses are tested per week?
Previously, we used the hypothesis testing approach every Monday from two to four hours. The goal was to make it a habit for product managers to test a new hypothesis, test a new solution, prepare a prototype, and conduct an in-depth interview. After we realized that it became a habit for everyone, we changed the system.
We did this in the form of an event on Friday, where everyone shared his impressions about testing the hypotheses. We attracted to this product not only from the Navigator, but also from Geoservices. We made a rating of employees, where each received stars for their research.
It was after a small team first took a design thinking course. It was an online course, but with an offline part that required the entire team to complete tasks. When we realized that this approach was popular inside the company, we began to independently conduct a course for product managers from Geoservices. We successfully implemented this initiative, and in the future it helped the internal growth of employees.
We conduct all analytical hypotheses in the task management system. Therefore, it helps us to analyze their number and apply further the calculations that were made for research. On average, in a quarter, we have dozens of different tasks for generating hypotheses, then confirming or disproving them. You can probably rate them by the hundreds.
And what about the failed experiments?
Case with parking. Now navigators are set up just to bring you to the final point and do not help, for example, to find a free parking space for a car. We decided to gradually get closer to this. First, we added a layer of traffic jams on which we showed where we could stand, then we added prices for parking, then payment for parking.
We wanted to make the product as profitable as possible for the user. But the task with the display of free parking spaces was difficult for us. We tested various hypotheses for tracking free parking spaces. But all the hypotheses were not confirmed. Now we decided to use this service in the micro route. Micro route - building a small circular route around the destination point so that the user does not go astray and finds parking.
An unsuccessful case for us is the experiment in which, when analyzing the metrics, we notice that people more often began to climb into the settings in order to disable the new feature. This happened when we replaced the familiar interface with another, more constructed, in our opinion. As a result, people began to use less navigator.
After testing, we were convinced that the most important information should be shown in one place, and not duplicated throughout the product.
What approaches do you use in predictive analytics?
We have a suggestion that is based on the user's past travels and offers automatically saved routes. This is machine learning in the product.
We also use the classical forecasting model. We distinguish components from the user's behavior - this is seasonality, time, including looking at the weekly cut, we take into account the criteria of the region and carry out the trend. Summing up all this, we get a forecast for the future.
Since we have two regions - Russia and Turkey, the indicators of the frequency of use of the product differ. For example, in Russia, the audience grows on weekdays, and in Turkey at the weekend. We also have forecasts of both important KPI metrics and those that can potentially become KPI metrics.
We use forecast data not only in analytics, but also in development. When we predict traffic jams on the route time, the trend for the next hour and a half is taken into account. This development was introduced through experiments, working with the audience.
For us, one of the main metrics is quality. Accordingly, we consider the relative error of this indicator and see how the indicator has changed after the introduction of the traffic jam forecast. We also consider it under various conditions. For example, the history of product use at rush hour on the roads or use on growing traffic jams. This is important because the situation on the roads is changing very quickly, and the navigator must be able to take this into account.
When we look at the data slice, we take into account the statistics not only for the whole day, but also take into account each individual slice, which can be very different from the positive dynamics of the whole day.
What is the model of monetization at Yandex Navigator?
Now geoservices at Yandex are separate business units. Therefore, we are trying to find good forms of monetization for Yandex.Navigator. We conducted several experiments, launched a couple of years ago pins along the route - this is the lightest format, it tells well about the location in which your business is located.
We are also looking for different formats, as it is possible inside the navigator to natively help the advertiser reach the motorist. In addition, we use such advertising formats as billboards, which hang in the product interface when following the route.
We have special projects that create some kind of emotion in a person. Together with the advertiser, we form a message about what a person is cool to do in the near future. For example, we had special projects with the use of additional voices in the navigator. Or, for example, for the centenary of Porshe, we replaced the arrow, the cursor that leads you along the route, with a Porsche model.
We try our best to suggest where to go. For example, now we are testing this format when the user has built a route, and we know that there is a MacAvto right on the road. If the time of the route is not very different, then we will offer him, with one button, to add Makavto to the script.
This is a cool format. Firstly, because he has high conversions, and, secondly, because he is super contextual.
What resources for self-education do you read?
I am helped by things that broaden my professional outlook and are not directly related to the area of ​​work. For example, the company IDEO, which are the main creators of research on the topic of design thinking in the world. They have a guide where they list what research they use with the Design Kit cases.
This is very similar to the approach to programming when new algorithms are invented. They are thought out by borrowing from things that already exist in life.
More product marketing reports available at the @epicgrowth Telegram Channel.
Epic talks is the Epic Growth project, where we communicate with teams about proven processes, testing of product hypotheses, analytics and much more. The project was filmed with the support of the AppMetrica analytical platform.
Team
Lyosha Gozhev, Senior Food Analyst and Misha Vysokovsky, Head of Yandex.Navigator
')
How is the product team Yandex. Navigator?
Inside the Navigator, we have several product teams, each of which has its own focus. Focus sets the direction for product development. For example, one team takes care of the route, the other focuses on the search script. Unlike a formed product team, the focus may change over time. Usually each team has a product manager, designer and developer.
We now have two analytics, so they work for several product teams. Mostly analyst tasks are divided into two types.
The first type is adhoc-tasks that need to be calculated quickly, or tasks requiring verification of code writing. The second type is research tasks in which you need to dive with the whole team. In such tasks there is no separation between the performer and the customer. The whole process of research tasks is very similar to traversing a growing tree: we generate hypotheses and then test them. During their refutation or confirmation new hypotheses arise.
Food experiments
Give an example of successful experiments?
We try to guess where the person will go and offer him the “destination suggest”. For example, the user opens the navigator, and the application offers to go on the most frequent routes. Now we have about 13% of the routes being built this way. This indicator is constantly growing. We are engaged in improving the predictive model, we look at how it affects and how much more convenient it becomes for the user.
Another example of a successful experiment is interface redesign using conventional electrical tape. Since we have a specific product, testing the Navigator while sitting at the table is a waste of time. At some point, we looked at the interface of J. Navigator and realized that there was a lot of duplicate information in it.
We thought: “Listen, the third screen on top is occupied by a huge black panel. Maybe we should throw her out and everything will be fine without her? ” The easiest option is to take the usual electrical tape and glue the top plate.
The first feedback, which appeared - is "I understand where to go, but it is not clear when I come." We took the scissors and cut them a little so that the time of arrival could be seen. We were surprised by the positive feedback. People quietly reached the place without information, which was sealed with black electrical tape, despite the fact that they were driving along an unfamiliar route.
How many hypotheses are tested per week?
Previously, we used the hypothesis testing approach every Monday from two to four hours. The goal was to make it a habit for product managers to test a new hypothesis, test a new solution, prepare a prototype, and conduct an in-depth interview. After we realized that it became a habit for everyone, we changed the system.
We did this in the form of an event on Friday, where everyone shared his impressions about testing the hypotheses. We attracted to this product not only from the Navigator, but also from Geoservices. We made a rating of employees, where each received stars for their research.
It was after a small team first took a design thinking course. It was an online course, but with an offline part that required the entire team to complete tasks. When we realized that this approach was popular inside the company, we began to independently conduct a course for product managers from Geoservices. We successfully implemented this initiative, and in the future it helped the internal growth of employees.
We conduct all analytical hypotheses in the task management system. Therefore, it helps us to analyze their number and apply further the calculations that were made for research. On average, in a quarter, we have dozens of different tasks for generating hypotheses, then confirming or disproving them. You can probably rate them by the hundreds.
And what about the failed experiments?
Case with parking. Now navigators are set up just to bring you to the final point and do not help, for example, to find a free parking space for a car. We decided to gradually get closer to this. First, we added a layer of traffic jams on which we showed where we could stand, then we added prices for parking, then payment for parking.
We wanted to make the product as profitable as possible for the user. But the task with the display of free parking spaces was difficult for us. We tested various hypotheses for tracking free parking spaces. But all the hypotheses were not confirmed. Now we decided to use this service in the micro route. Micro route - building a small circular route around the destination point so that the user does not go astray and finds parking.
An unsuccessful case for us is the experiment in which, when analyzing the metrics, we notice that people more often began to climb into the settings in order to disable the new feature. This happened when we replaced the familiar interface with another, more constructed, in our opinion. As a result, people began to use less navigator.
After testing, we were convinced that the most important information should be shown in one place, and not duplicated throughout the product.
Analytics
What approaches do you use in predictive analytics?
We have a suggestion that is based on the user's past travels and offers automatically saved routes. This is machine learning in the product.
We also use the classical forecasting model. We distinguish components from the user's behavior - this is seasonality, time, including looking at the weekly cut, we take into account the criteria of the region and carry out the trend. Summing up all this, we get a forecast for the future.
Since we have two regions - Russia and Turkey, the indicators of the frequency of use of the product differ. For example, in Russia, the audience grows on weekdays, and in Turkey at the weekend. We also have forecasts of both important KPI metrics and those that can potentially become KPI metrics.
We use forecast data not only in analytics, but also in development. When we predict traffic jams on the route time, the trend for the next hour and a half is taken into account. This development was introduced through experiments, working with the audience.
For us, one of the main metrics is quality. Accordingly, we consider the relative error of this indicator and see how the indicator has changed after the introduction of the traffic jam forecast. We also consider it under various conditions. For example, the history of product use at rush hour on the roads or use on growing traffic jams. This is important because the situation on the roads is changing very quickly, and the navigator must be able to take this into account.
When we look at the data slice, we take into account the statistics not only for the whole day, but also take into account each individual slice, which can be very different from the positive dynamics of the whole day.
Monetization
What is the model of monetization at Yandex Navigator?
Now geoservices at Yandex are separate business units. Therefore, we are trying to find good forms of monetization for Yandex.Navigator. We conducted several experiments, launched a couple of years ago pins along the route - this is the lightest format, it tells well about the location in which your business is located.
We are also looking for different formats, as it is possible inside the navigator to natively help the advertiser reach the motorist. In addition, we use such advertising formats as billboards, which hang in the product interface when following the route.
We have special projects that create some kind of emotion in a person. Together with the advertiser, we form a message about what a person is cool to do in the near future. For example, we had special projects with the use of additional voices in the navigator. Or, for example, for the centenary of Porshe, we replaced the arrow, the cursor that leads you along the route, with a Porsche model.
We try our best to suggest where to go. For example, now we are testing this format when the user has built a route, and we know that there is a MacAvto right on the road. If the time of the route is not very different, then we will offer him, with one button, to add Makavto to the script.
This is a cool format. Firstly, because he has high conversions, and, secondly, because he is super contextual.
About education
What resources for self-education do you read?
I am helped by things that broaden my professional outlook and are not directly related to the area of ​​work. For example, the company IDEO, which are the main creators of research on the topic of design thinking in the world. They have a guide where they list what research they use with the Design Kit cases.
This is very similar to the approach to programming when new algorithms are invented. They are thought out by borrowing from things that already exist in life.
More product marketing reports available at the @epicgrowth Telegram Channel.
Source: https://habr.com/ru/post/455062/
All Articles