"How to measure the fun of the game" and other frequently asked questions about application analytics

Every month we have an AMA (ask me anything) session on reddit - we answer any questions about application analytics. We decided to select the most interesting questions and provide answers to them here. What types of games do you need analytics for? How to conduct experiments? How to increase retention? Answers to these and many other questions - under the cut.


[-] jakkarth
Do I need analytics for all types of games? I admit that it is useful to use analytics in such large and complex games as World of Warcraft, but is it necessary in casual mobile games that people play for 5 minutes? Where is the critical mass, after which an analyst is required? Are there any genres or types of monetization that are harder to analyze than others?
devtodev:
My opinion - analytics is important regardless of the size of the game. Of course, the larger the project, the more complex analytics it requires. When you are preparing the game for launch, you still do not know its future scale, but this does not mean that you do not need an analyst.
By the way, do not underestimate the volume of casual mobile games, where people play from time to time. Read the post of Paul Murphy, the founder of Dots, for example. I imagine how big and complex analytics is built in their company.
And answering your last question, I will say that the depth and complexity of analytics depends more on the scale of the project than on the genre.

[-] seardluin
What metrics would you recommend to use in the RPG game for PC, in which there are no built-in purchases, but is it possible to play online? Would you analyze the offline game, or just the online component?
devtodev:
I think that you are in any case interested in the loyalty of users, for example, to accumulate a user base for subsequent games. Therefore, I would recommend tracking both online and offline metrics.
What metrics to use?
First of all, audience metrics: DAU, MAU, loyal to DAU (minus one-day users).
Then I would recommend using retention and activity metrics of players: retention from 7 to 90 days, sticky factor, number of sessions per user.
')
[-] Novamonk1
How can you tell if users enjoy the game?
[-] koorb How do you measure a custom fan?
devtodev:
Metrics such as pleasure or fan simply do not exist, and therefore we cannot measure them. We can only somehow bring them closer, approximate the existing metrics:
Retention (1 day, 7 days, 30 days and above).
Sticky factor. It is calculated as the ratio of DAU to MAU. Good metric talking about the regularity of user logins. It has a strong correlation with the income of the game.
The number of sessions per day. It is believed that for a good healthy gaming mobile project the optimal value is 3 sessions per user per day. In games with a longer session (say, RPG) - two sessions. In games with a short session (casual games, runners) - 4-5 sessions.
The average session length and the average time spent by the user in the game per day (session length * number of sessions).
K-factor as an indicator of virality. If the user likes the game, he is more likely to tell friends about it. Usually the K-factor is calculated as the average number of invitations sent, multiplied by the average conversion from invitation to registration. However, for mobile projects it is difficult to track the user base covered by the invitation, and therefore the K-factor is calculated as the ratio of the number of new organic users to (DAU - New users).
I would not recommend these metrics to be used to compare the pleasure that users get from two different games. But they are perfect for measuring changes in the attitude of users of one particular game to changes in it.

[-] gruegoo
What should I compare the values ​​of my metrics with? I want to compare DAU / MAU indicators and the percentage of people paying with other indicators in the industry, but I don’t know where to get the data. Speech is not only about these metrics specifically, but generally about the search for samples for comparison.
devtodev:
I would also like to have an inexhaustible source of benchmarks throughout the industry, it would be really useful. But in practice, you just have to look for information on the Internet yourself.
A good resource is Quora.com, a question and answer service. There you can ask a question about the values ​​of metrics in a particular genre, in a particular country. The audience of this service is happy to share information. Here is an example .
I also recommend subscribing to devtodev blogs and other analytic services. There you can often find articles with data that can be used for benchmarking. Here are some examples: a GameAnalytics article and a devtodev webinar entry.

[-] Jerseywhat
How do you think, how to organize experiments?
What role should analysts play in this process? Decide what kind of experiments to run, set priorities?
If a team wants to run several experiments at the same time, will you try to somehow analyze their results, or will you offer to divide the experiments in time? Run multiple experiments in parallel, or run one experiment with many different options (for example, group A: high complexity, low costs, group B: medium complexity, low costs, and so on)?
How do you work with statistical significance? What is better, wait until you get a large enough sample to make the result significant, or choose one of the options as quickly as possible, without waiting for the end of the experiment?
devtodev:
I think the task of the analyst is not to offer options for testing, but to provide a sufficient amount of information to the game designer so that he himself suggests various hypotheses. And subsequently, it is the analyst who must evaluate the result.
I am in favor of running unitary tests. Yes, it will take more time, but the results will be more reliable. If you run multiple experiments simultaneously, there is a high risk that the sample for each option will be too small, and the results may be erroneous.
This is a question of the scale of the project. Usually, small projects want to make decisions and make changes instantly, and large projects need more time. As an analyst, I would not recommend making important decisions in a hurry. But perhaps this is just a professional deformation from the fact that you usually have to deal with large projects.

[-] bimbompaul
I know there are two ways to calculate retention - classic and rolling. What is the difference? Which one is better to consider in practice?
devtodev:
Good day!
In the case of classical retention, the user is considered to be retained on the first day if he entered the application one day after the first visit. In rolling retention, the user is considered to be retained on the first day if he entered the application one day after the first visit or later. Therefore, rolling retention values ​​are always higher.
Both retention serve the same purpose — measuring user retention, and they are strongly linked to each other. It is difficult to imagine that many users will not log in for the first few days, and then start using the application regularly. The higher the classical retention, the higher the sliding, and vice versa.
Here is an example of classic and rolling retention values ​​for different projects. If your project is based on long-term retention and user communication, I would recommend using sliding retention. If you need benchmarks and benchmarks, then it is better to use classical retention, simply because most of the benchmarks on the Internet are given for the classical one.

[-] cairmen
Great topic for the question-answer column!
In your opinion, with which metrics can one find the simplest ways to increase income and retention?
devtodev:
By themselves, metrics will not increase your income and retention, but they will allow you to generate ideas on how to achieve this.
Let's discuss. The key to successful monetization is long-term retention (the longer a user is with you, the higher are his chances of payment, and the average of his check). The easiest way to increase long-term retention is to optimize short-term retention.
Well, the easiest way to increase short-term retention is by optimizing the first user session (FTUE, first time user experience).
Another reason for optimizing FTUE is the fact that it was during the first session that the fundamentals of user activation were laid, which, in turn, leads to success in monetization.
So try to optimize the first user session:
Take a look at your product through the eyes of a new user: what does he feel, what does he think about after one, two, three minutes in your service?
integrate user events into your tutorial (our practice shows that many users leave the tutorial just for technical reasons, and your task is to understand and optimize)
show the benefits of your product over all others
intrigue the user, let him know what will only be more interesting.
To measure success, use the following metrics:
the percentage of users who successfully passed the tutorial;
1-day retention (both classic and rolling);
the percentage of conversion in the first purchase.

[-] aschearer
What three metrics would you single out among those that best describe the state of the game, and why?
devtodev:
Good day! Let's look at the game as a model - at the entrance new users, at the exit money. Therefore, if you select three metrics, it will be:
new users;
the percentage of paying users;
average income per paying user (ARPPU).
If I know these metrics, I can count key indicators and compare businesses with each other. And if you allow me to add a fourth metric, I would add retention.

[-] BikerMouseFromMars
What books would you recommend to start studying game analytics?
devtodev:
Most of all I like this book .
I also recommend the book “The Goal” by Eliyahu Goldratt. It is not about games, but in general about finding bottlenecks in business, and this is relevant for everyone.

[-] svero6
I have some big questions at once.
What are the standard metrics for MMO games? And how much do the metrics of hit games, medium games (for example, platformers) and unsuccessful games differ from each other?
If you have an in-game store selling virtual goods for virtual currency, how will you optimize the selling price? How do you understand that this or that price is too high? Or too low? Usually AB-tests are used for such purposes, but in online games users communicate with each other a lot, and if you experiment with prices, it will not go unnoticed and will break your experiment. How else can you optimize the price, except how to change it and see what happens?
Finally ... How to understand what volume of goods to give to users for free? For example, in Hearthstone, you earn gold by winning matches, and you can spend this gold to buy sets of cards or other goods within the game (instead of buying them for real money). In other games you accumulate virtual currency, completing missions, and so on. How to understand how much to give gold to users? How to optimize it?
devtodev:
Thanks for the interesting questions!
> What are the standard metrics for MMO games? And how much do the metrics of hit games, medium games (for example, platformers) and unsuccessful games differ from each other?
First of all, for MMO games, I would recommend using a standard set of metrics .
I also recommend paying attention to the in-game currency, the dynamics of its spending and savings. Do not let the average balance of currency in the accounts of players grow, otherwise the day will come when you have to raise prices.
Regarding the difference between hit, medium and unsuccessful games, I recommend the GameAnalytics article , in which you can see how the metrics differ.
> If you have an in-game store selling virtual goods for virtual currency, how will you optimize the selling price?
Good question. You are right, the first idea is to conduct an AB test, but this is not possible in projects where users communicate with each other a lot.
I would recommend using ABC / XYZ analysis for your virtual goods. This will allow you to highlight the categories of products where you can easily change the price (subsegments BY, CY, BZ, CZ).
If we talk about the subsegments with the letter A, then it is hard to change something when the game is already running. Therefore, it is better to form and fully calculate the consumer basket before launch. Say during beta.
> How to understand how much product to give to users for free?
This is more a game designer, rather than an analytical question. From my position I can only recommend a few links to materials that can help you find the answer:
www.dubitlimited.com/blog/2010/12/31/design-a-social-game-economy-how-much-should-a-virtual-t-shi-html
www.quora.com/How-do-you-set-prices-for-virtual-goods-in-social-games
Freemium Economics (Eric Seufert)
Virtual Economies: Design and Analysis (Vili Lehdonvirta, Edward Castronova)
Free-to-Play: Will Luton
A massively multiplayer online game case study, runescape (Tanla E. Bilir.)

[-] thealchemistbr
I work on a game in which players need to build pyramids from numbers in a special way. The game has three levels of difficulty. What metrics should be used? How can I increase retention? And how do I choose a monetization model based on this information? Thank!
devtodev:
Good day!
> What metrics should be used?
If your game is not an MMO with a complex economy, a standard set of metrics is enough for you. I would recommend paying particular attention to the session metrics (the number of sessions in general and the player, the average session length). It seems to me that this is important for casual gameplay. If I'm wrong, let me know.
> How do I increase retention?
There are many ways to increase retention. One of the most effective is tutorial optimization. Teach your users to play, show them the benefits of your game. A good tutorial is a good one-day retention, which means good long-term retention, and therefore money.
I can also recommend daily quests, regular tasks. Well and nobody canceled deep-linking with push-notifications.
Here is our webinar about retention, there are more tips on how to increase this figure. how to increase it.
> And how can I choose a monetization model based on this information?
Still, the monetization model is usually chosen before you begin collecting data. For casual games, the most popular and effective model is, of course, free-to-play. If you run a f2p game with a long-term hold, you have every chance to succeed. See , for example, how Clash of Clans did it.

Conclusion
This is a big part of the questions we have come to love, but, of course, far from all that we can answer with the help of analytical tools. If you are interested in something else, we will be happy to answer in the comments or personal messages.