Blitz with Ilya Krasinsky: how to shoot bad hypotheses, why to dismiss the product and how to grow in a minimum of actions?
CEO Rick.ai Ilya Krasinsky answered the questions faced by product managers in the Q & A session format at the Epic Growth Conference.
See the decoding under the cut.
')
Any feature always lowers metrics. We automate this or that scenario of human behavior. If the percentage of people who understand the benefits of the feature and used it is almost zero, then the value of the feature will be minimal.
There are disadvantages after the launch of a new feature in the product: the base code grows, new bugs and defects appear, the first session is more difficult for users, and activation is more difficult.
There is a method of evaluation. You take a stream of users who will use this feature. So you can assess how much the conversion and income from a paid user will change. Then you will be able to calculate how much money a new feature can bring you.
Very often, the development of the company hinders the lack of funding. In this case, the business becomes not an asset for you, but a suitcase without a handle, which is pitiful to throw, but also hard to carry.
Accordingly, in this situation, squeeze the maximum until you shoot this “dead horse”, because next time it will take you two to five years to get closer to your current point of development. Determine what skills and experience you can bring to yourself and do not be afraid to get rid of the ballast.
The main problems of current analytics:
A large number of companies already have a division of Big Data. The vast majority of these units are engaged in data storage, in the smaller - in the compilation of algorithms. There is usually a technological gap between top management, product managers and data science analysts. Business guys often do not understand what question to ask analytics.
Basically, these data are not consistent, that is, they already contain errors at the collection stage.
Analytics is very fragile, it is very easy to break. Accordingly, the key question is whether you have a built-in monitoring system of robotic analytics.
The main barrier to the transition to robotic analytics is that the data does not accumulate in the systems you use and, accordingly, give incorrect numbers. Therefore, any conclusions and management decisions will also lie.
Until this problem is solved at the level of integration and data flow, everyone will probably be cutting their Death Star, believing that it works. I have already filed five such systems in my life, and each time the developers found bugs and defects in them.
Two types of machine learning must be distinguished:
The latter type is much more used. But, unfortunately, in practice it is very poorly implemented. These presentations do not go well into a working product. Accordingly, the main problem of machine learning is that people see a black box at the exit.
I believe that a person can not process such information flows for a long time. I see what we're all going this way: either there will be black boxes, like attribution models that Google does, or some system that will analyze the data and explain to the person how she analyzed what is in this slice ( , domain, conversion).
You already have them: “Google spreadsheets” or Excel.
Most of the hypotheses can not change any metric, can not do anything good to the user, they need to shoot. And out of 50 hypotheses, if you leave seven, you have a chance of success.
In 2019, it is clear that people still feel worse than a calculator. But it seems that a person knows how to invent non-standard ideas.
The easiest way: Talk with ten industry professionals in the networking format at the conference. You will receive a list of fifty questions. Leave those questions that you like, and you will get some kind of framework.
How is it in our team:
- A person must have a high level of energy. If a person has a low level of energy, then the whole team will be toxic.
- A person must be systemic and with the experience of reflection. Developing the skill of systematization is very expensive and time consuming. It is checked quite simply: ask a person a question about his previous work experience, including a negative one, and what conclusion he made from this experience.
Approximately 50% of people say: “Thank you, great question! I'll go think about it. " This means that in the last year, when this situation occurred, they did not do this work. They have no such habit.
- A man should not be afraid. During the work it is necessary to make a large number of decisions, most likely, the product will be wrong. It is important that he was not afraid to do this.
Trigger analysis. You take a user segment, watch all user sessions and a chain of events. Divide people into two groups: those who are in retargeting, and those who are not.
In practice, we must understand that we never have the task of measuring something extremely precise. Often this is just pointless. If your investment in retargeting is less than the amount of work that I just described, then the work itself on analyzing retargeting will be more expensive than just doing it.
You need an accurate attribution model. Let us check with the concepts: it is not necessary to accurately attribute one or another income to any advertising campaign. We have only four management decisions:
- If his model of the world is much inconsistent with reality.
- If his hypotheses are weak and poorly correlated with our users.
- If you do not like to communicate with users.
- If you do not like to do corridors, kastdevy.
- If not testing your hypothesis.
- If using an irrelevant set of tools.
- This means that he will be greatly mistaken in his conclusions, does not want to learn how to do things right, and simply does not follow the latest frameworks that are occurring in the industry, which means that he has fallen off.
An aggravation happens at the end of the year. At the end of the year, people remember the goal they set for themselves.
The meaning of this - you must be able to lose. You need to repeat yourself: there were a lot of experiments, so I just didn’t take something into account and did not understand. They changed the unit-economy, changed the approach, raised the conversion, but this does not mean that the project will be all right.
Encourage support and care within the team. One of the skills that I am developing in myself right now: how to explain the product manager, designer, marketer, analyst, that they did everything wrong, but at the same time, so that they do not have hands and they went to work the next day with the words: "Ok, the seventh time we will redo everything, and we will succeed."
I believe that they are quite a lot. For example, Ultimate Guitar, Skyeng, RealtimeBoard. For the success of such companies are not only the first persons who are in full view, but also the performers who do a tremendous job every day.
It's cool to be friends with them. It's just a free train on which you get new ideas, books, and frameworks. Therefore, it seems to me that surrounding myself with such a list of people is one of the important tasks.
More practicing narrow product skills at Epic Workshop Day .
See the decoding under the cut.
')
What is an effective method for assessing the viability of a feature?
Any feature always lowers metrics. We automate this or that scenario of human behavior. If the percentage of people who understand the benefits of the feature and used it is almost zero, then the value of the feature will be minimal.
There are disadvantages after the launch of a new feature in the product: the base code grows, new bugs and defects appear, the first session is more difficult for users, and activation is more difficult.
There is a method of evaluation. You take a stream of users who will use this feature. So you can assess how much the conversion and income from a paid user will change. Then you will be able to calculate how much money a new feature can bring you.
At what point do you say “enough is enough”? Or do you need to jump and try until the lack of finance stops my startup?
Very often, the development of the company hinders the lack of funding. In this case, the business becomes not an asset for you, but a suitcase without a handle, which is pitiful to throw, but also hard to carry.
Accordingly, in this situation, squeeze the maximum until you shoot this “dead horse”, because next time it will take you two to five years to get closer to your current point of development. Determine what skills and experience you can bring to yourself and do not be afraid to get rid of the ballast.
When, in your opinion, more than 50% of companies (at least IT) will switch to robotic analytics by analogy with Rick.ai? What are the main barriers to date?
The main problems of current analytics:
A large number of companies already have a division of Big Data. The vast majority of these units are engaged in data storage, in the smaller - in the compilation of algorithms. There is usually a technological gap between top management, product managers and data science analysts. Business guys often do not understand what question to ask analytics.
Basically, these data are not consistent, that is, they already contain errors at the collection stage.
Analytics is very fragile, it is very easy to break. Accordingly, the key question is whether you have a built-in monitoring system of robotic analytics.
The main barrier to the transition to robotic analytics is that the data does not accumulate in the systems you use and, accordingly, give incorrect numbers. Therefore, any conclusions and management decisions will also lie.
Until this problem is solved at the level of integration and data flow, everyone will probably be cutting their Death Star, believing that it works. I have already filed five such systems in my life, and each time the developers found bugs and defects in them.
My advice: duplicate the data so that you have different analytics systems so that you can verify the numbers with each other. One system is a very unreliable thing, mistakes happen very easily.
What are the prospects for ML in predictive analytics?
Two types of machine learning must be distinguished:
- Compiled in the Python programming language.
- Compiled using a PowerPoint presentation.
The latter type is much more used. But, unfortunately, in practice it is very poorly implemented. These presentations do not go well into a working product. Accordingly, the main problem of machine learning is that people see a black box at the exit.
I believe that a person can not process such information flows for a long time. I see what we're all going this way: either there will be black boxes, like attribution models that Google does, or some system that will analyze the data and explain to the person how she analyzed what is in this slice ( , domain, conversion).
How likely is the emergence of tools to test hypotheses before changes are introduced in the product?
You already have them: “Google spreadsheets” or Excel.
Most of the hypotheses can not change any metric, can not do anything good to the user, they need to shoot. And out of 50 hypotheses, if you leave seven, you have a chance of success.
In 2019, it is clear that people still feel worse than a calculator. But it seems that a person knows how to invent non-standard ideas.
What questions to ask the product at the interview?
The easiest way: Talk with ten industry professionals in the networking format at the conference. You will receive a list of fifty questions. Leave those questions that you like, and you will get some kind of framework.
How is it in our team:
- A person must have a high level of energy. If a person has a low level of energy, then the whole team will be toxic.
- A person must be systemic and with the experience of reflection. Developing the skill of systematization is very expensive and time consuming. It is checked quite simply: ask a person a question about his previous work experience, including a negative one, and what conclusion he made from this experience.
Approximately 50% of people say: “Thank you, great question! I'll go think about it. " This means that in the last year, when this situation occurred, they did not do this work. They have no such habit.
- A man should not be afraid. During the work it is necessary to make a large number of decisions, most likely, the product will be wrong. It is important that he was not afraid to do this.
How to measure the incremental effect of retargeting?
Trigger analysis. You take a user segment, watch all user sessions and a chain of events. Divide people into two groups: those who are in retargeting, and those who are not.
In practice, we must understand that we never have the task of measuring something extremely precise. Often this is just pointless. If your investment in retargeting is less than the amount of work that I just described, then the work itself on analyzing retargeting will be more expensive than just doing it.
You need an accurate attribution model. Let us check with the concepts: it is not necessary to accurately attribute one or another income to any advertising campaign. We have only four management decisions:
- turn off the process, do not converge at all; we spend too much, get almost nothing;
- you can slightly modify;
- do not touch;
- amplify.
Why would you fire a product?
- If his model of the world is much inconsistent with reality.
- If his hypotheses are weak and poorly correlated with our users.
- If you do not like to communicate with users.
- If you do not like to do corridors, kastdevy.
- If not testing your hypothesis.
- If using an irrelevant set of tools.
- This means that he will be greatly mistaken in his conclusions, does not want to learn how to do things right, and simply does not follow the latest frameworks that are occurring in the industry, which means that he has fallen off.
You conduct experiments, 95% of failures, little success, constantly decay and pain in the head. How to be?
An aggravation happens at the end of the year. At the end of the year, people remember the goal they set for themselves.
The meaning of this - you must be able to lose. You need to repeat yourself: there were a lot of experiments, so I just didn’t take something into account and did not understand. They changed the unit-economy, changed the approach, raised the conversion, but this does not mean that the project will be all right.
Encourage support and care within the team. One of the skills that I am developing in myself right now: how to explain the product manager, designer, marketer, analyst, that they did everything wrong, but at the same time, so that they do not have hands and they went to work the next day with the words: "Ok, the seventh time we will redo everything, and we will succeed."
The coolest food teams in Russia?
I believe that they are quite a lot. For example, Ultimate Guitar, Skyeng, RealtimeBoard. For the success of such companies are not only the first persons who are in full view, but also the performers who do a tremendous job every day.
It's cool to be friends with them. It's just a free train on which you get new ideas, books, and frameworks. Therefore, it seems to me that surrounding myself with such a list of people is one of the important tasks.
More practicing narrow product skills at Epic Workshop Day .
Source: https://habr.com/ru/post/457338/
All Articles