Metrics vs. Experience

This publication is sometimes a free translation of the article by Julie Zhuo, product designer on Facebook. Enjoy reading.

If a few decades ago you wanted to do something unique, you would sit down, take a deep breath, close your eyes and turn to an oracle called “intuition” with a prayer.
')
But, as our ancestors once tamed the power of fire, so we learned to use the capabilities of metrics and analysts. We are no longer wandering in darkness, wondering, "Does anyone really read these emails sent every two days?". I even wonder how many people tried to find an answer to this question until this point, before the onset of the future created by all of us. After all, now we simply plunge into the treasury of numbers and get the answer out of it.

Alas, despite the light gained by us, existential shadows remain in the corners. I heard them in discussions in the halls of the office, in mumbling over the glass after work, I saw typed capital letters in the threads of style discussions that wandered around the network:

“Do we just do this to get a metric?”
“How can we balance the numbers and do something worthwhile?”
And my favorite: “Do you, the data managers, really care about the users and the UX?”

Oh! Strong words and burning accusations!

Maybe you want to talk productively about metrics and positive experiences? That's what I know.

Do not create a framework in the style of "Metrics vs. Experience"

Comparing to fans of some big-budget blockbusters about superheroes and starting a conversation with the assertion that something is better, you are doing wrong.

Comparing Metrics and Experience is the same as comparing “carbohydrates and healthy foods”. You should not use such comparisons if you want to start a discussion about healthy eating.

Being able to measure gives you an understanding of what people are doing in your product. If you do not want to live, trusting in fate, then possessing a large amount of information is a good thing. Of course, you should be able to distinguish which information is important and which is not, but the statement that the very fact that you have access to more information is wrong and unjustified. Metrics are not evil.

In addition, if you are doing something that people find valuable, your actions should be reflected in the growth and success of the product. You cannot argue that your update introduced something better if people did not change your use of your product. And vice versa, if you made any changes that led to a smaller audience, you cannot “brush off” fairly clear facts and evidence that you have spoiled something.

And finally, the third reason why metrics are so valuable is that they help to rally the team around something clear and tangible, so that people can see their own results. From a purely material and technical point of view, it is difficult to manage 50 people with the installation “we must create something cool!”. Of course, anyone can wave their fists after a fight and shout “Yes! We did something amazing! This is what we wanted to achieve! ”, But when it is Monday morning and team A appears in the office and begins to praise themselves, team B can react in approximately this style:“ Um, no, this is actually a piece of shit ”. What is going on? How to clearly define what is this “something cool”?

One way to solve this problem is to build a hierarchical ladder within the team. You can designate a specific person (or list of persons) in the company, so that they act as “judges”, what is “cool” and what is not. If you prefer to work without a hierarchy, then another method available to you is the definition of a measurable goal. “The super high quality of the product means that 50% of users will turn to this“ feature ”again within a week.” Now Team A and Team B know exactly what they are going to every day and how close they are to this goal.

Summarizing, we can say that with this approach both the UX and the metrics are useful and not limited to the eternal struggle with each other. And make sure you stick to that point of view.

Bad decisions are made in the name of "improving performance"

Also, as you can overeat donuts and call carbohydrates "bad", so statistical indicators can be used to justify bad decisions. This happens because not everything you can measure is really worth measuring. It can also happen because you did not become familiar with the dynamics and viewed the statistics for a single period. Often you need a whole set of metrics to get a really complete picture of what is happening.

If you accidentally chose the wrong object to get the metric, based on which you are going to move on and develop your product, you may end up with something truly “harmful” in terms of UX. Consider a few examples:

a) “Initially, the clickthrough rate for this feature was 2%. When we made changes, the rating went up to 5%! Hooray!".

What is the problem? The increase in functional clicks does not tell us that the UX has actually been improved. And what if I changed all the links on the site to "click here to earn 250 bucks"? Damn it, the clicks really will crawl up! But in the end, people will realize that I’m not really giving them that 250 bucks and get mad at me. They will cease to follow my links, remove the application and slap me a star in the store with the posts "FUUUUUU". And yes, it is in capital letters. My business will be ruined, my life will turn into shit. The end.

b) People used to spend 5 minutes on my application. Now, after the launch of the last entered function, they began to spend only 3 minutes. Uh-uh, why are you?

What is the problem? Is the time spent in your application actually an important indicator to keep track of? Everything is relative. If your target product is content, then yes, there are reasons for increasing the usage time of your product, because the more a user reads / watches / listens, and the more time he spends, then accordingly, the more likely he will find for yourself something interesting and useful. But if your application is designed to electronically sign documents, then this approach is incorrect. You should better track some other indicator, for example, the number of actual uses of the functionality of your application, because it helps people sign documents faster. And, in fact, if a person is able to complete this part of his work faster (sign a document), then he will clearly like it, i.e. reducing the time to use the application while maintaining the number of operations performed over time will have a positive impact on your product.

c) "Initially, our application with photos of cats and memes was used more in Illinois, but now we have more users in Ohio."

So what's the problem? Specifically, it does not matter and not such information can not be a metric to which one should pay attention and try to somehow influence it.

Of course, there are important things that cannot be easily and accurately measured.

If we could read the user's mind, then we could theoretically design the perfect UX. Unfortunately, we are all not Gina Gray ( telepath from the X-Men universe, note ), so we have to measure everything we can, do everything to build reasonable assumptions about what people need. But the volume of measured data has its limits and it is important for us to remember this. Just the statistics of what people do in your product cannot tell you about such things as:

• The degree of loyalty, hatred or indifference to your product because of some specific feature of it;
• Does collecting statistical data increase or decrease people's confidence in your product over time;
• How simple and easy your product is perceived by users;
• How people see your product compared to similar offers on the market;
• What things users would like to change, add or leave in this form, as it is;
• What will be the level of interest in your product over time.

You can get answers to some of the above questions by conducting quality research, surveys, and interviews, but none of these tools are perfect (remember the Brexit predictions?). And even in cases where the opinion of the masses can be accurately measured (for example, the degree of trust in the brand), it is difficult to understand exactly what actions and changes led to a similar reaction (did the logo become the cause? people to my brand improved?).

Unable to accurately measure the positions described above, the metrics in these cases let us down. Note the following:

Understanding the cost of complexity: every time you add a new function to your application, it’s likely that the metrics you are tracking show a positive trend (after all, until this point, nobody used X functionality, and now all more and more people are turning to it. And it seems that this does not lead to a decrease in the use of the functional Y or Z, so in general it looks like a victory). However, if you continue to add new features, at some point your product will become bloated. Then, suddenly, your competitor will go vertically up due to the fact that he will simply have implemented the Q functionality, which the audience likes. This is the paradox of choice and the size of cognitive effort. And we just have not understood how to measure this accurately.

Understanding the power of brand: when Apple or Nike launches its new product, many tend to buy it without even doing any analysis, simply because they have had positive experiences in the past this brand. But the same thing will not work if there is any upstart under the brand “Pear” (pear) or “Sike” with an equivalent product. At the general level, we all know and understand this. Nevertheless, it is difficult for us to assess the strength of the brand and turn it into a number that we can monitor daily. It is difficult to understand which of the thousands of actions will affect the company's image, what will be the benefits and losses from the adoption of a decision.

The power of big bets: no, metrics cannot tell you what bold action you need to take in order to win in the future. Imagine 2008, when smartphones just started to appear. Looking at the metrics of your site you would see a tiny piece of traffic coming from mobile devices. You might have come to the conclusion that it is not worth investing in optimization for mobile devices, because it is too small part of your audience. Today we understand the foresight of those who made a bet on the mobile segment and who are now reaping the laurels. No examination of current user behavior can say for sure in which direction you need to jump. Strategic, long-term planning still requires the same thing as always: trust your gut.

Some rules for clean metrics

Here are a few of my biggest lessons learned during the search, thanks to which you can build a more or less disciplined tactic of setting the right goals to measure.

When people choose something in the market, they come up and watch. Do not try to reach a huge number of people with your product or a specific function (the results may be distorted by aggressive promotion). Find out better if your product correlates with something valuable, because that is what will tell you whether the people you were trying to like are really interested in the results of your work in order to come back and use it again.

To optimize growth, you need to understand your “funnel” ( apparently, it means sales funnel, approx. ). In order for people to become regular users of your product, they must go through a bunch of obstacles. First, they must be aware of your product. Secondly, they must be interested enough to try it. Thirdly, they must pass the conversion (download the application, fill out the form, confirm the email address, and so on). Fourthly, they must take a lot of actions in your product in order to understand how it can be valuable for them. Fifth, they must remember it in order to return. At each of these stages you will lose users. If you can track and measure losses on each of them, you will have the opportunity to focus your efforts on making your “funnel” less full of holes. You can find out which indicators are really important and focus on them. Yes, it is very tempting to keep track of everything that is possible (because you have such an opportunity). You have a panel with numbers and you would like all of them to be "green." But it must be admitted that most of these data do not matter and that only a small part of them is truly important. Do not waste time talking about unimportant things, ignoring the fact that some really important indicators go up or down.

Use the magic wand trick to identify the best tracking score. Just ask yourself: “if I could wave a magic wand, after which I would cease to know anything about my audience, what would I like to learn first of all in order to make my application successful?”. Even if your answer is not something that can be measured, it will be useful and will create a starting point for work.

Do not just take metrics for the goal, not understanding them. The goals that will be the main for you and your team should be bought. Do not set nominal tasks. Put a certain meaning in them and imagine what they will be prompted for. There are situations where, it seems, a good solution does not affect the metric in any way. On the other hand, there are situations when the indicators are growing, but was the decision made really good? If so, then does the metric (or a set of them) actually work to track something really necessary?

There is also data that can be called a "countermeasure". If you look at the numbers that show a good result, ask yourself: “with what other information can I get to convince myself that these results are not as good as they seem?”. And this information is called a counter metric and every successful metric must have it. For example, look at the clickthrough rate, and how quickly they show a rebound. Or on the number of sales in comparison with the number of returns or cancellations. It is much better to be paranoid in terms of data interpretation, because this way you can quickly catch mistakes and correct your own strategy. Do not fall into the bias trap, where you are simply looking for signals to confirm your own expectations.

Use qualitative research to get answers. Use quantitative data that tells you what people have done in the application, paired with qualitative research that gives you an idea of ​​people's emotions and feelings. Conduct usability tests, use focus groups, run surveys and find out what is behind the behavior that you observe.