Custom game research
In January, as part of our program , Management of Gaming Internet Projects, Natalia Sprogis, a visiting expert (Head of UX Research at Mail.Ru) , held a master class on “User Research in Games” .
In this article we will answer the questions that were discussed at the master class:
- Why do game developers need UX research?
- What questions do they help answer?
- At what point in the development should they be used?
- How to organize a study?
')
Many experts from the game industry believe that research is not needed, or only big companies need it. Usually they give the following arguments:
- Why do we need research: we have experts who understand what to do and how.
- Experts are very good, but all that they have invented should be checked on living people. Why?
- Because you can come up with a great game, but present it to the wrong audience;
- Because you can come up with an excellent game, but to embody it in such a way that your audience will not understand it.
We have testers, so we don’t need research.
Testers check how well the game fits your idea, but they can't test how good your idea is.
We have beta testing with players, why do we need research?
First, the beta tests focus on the technical component.
Secondly, the most active and advanced players participate in beta. You get feedback and bug reports from activists and hardcore players, but you still don’t know why a regular player left you.
Why do everyone need UX research?
1. They help to understand a lot about your game. For example:
- Why do players leave after the first hour of the game?
- Why is the new quest a very low percentage of passage?
- Why is your game not like casual?
- Why do you play only non-paying users?
“Let's say you fall off part of the players after the 5th level. You need to understand why this is happening. Perhaps they are infuriated by the boss or the level is too difficult or boring. The user is tired and uninteresting to him. ”
2. They help to see the problems of gameplay and usability. During the study, you will be able to trace the user's behavior and use this knowledge in the future to correct errors.
3. They help you get to know your audience. Perhaps you imagine it completely differently. It happens that the development team is shocked when it meets their players for the first time on a test.
When should research be applied? There are different types of research suitable for each stage of development. They all answer different questions.
Possible issues at this stage of development:
• At what age of players should we be better oriented?
• What do players like about competitors? And what mistakes do not need to be repeated?
• Do we have chances to hook casual players?
• What features should be developed in the first place, and what can wait?
Consider a few examples of research at the stage of prototyping.
There is a certain game genre. The developer wants to make a game in this genre, but he does not have a clear vision of what it should be. Our task is to conduct research in order to understand what the players love in this genre, as well as to highlight groups of players according to their behavior and expectations from the game. To do this, we first conducted six focus groups. We collected 6-8 people of a certain profile and talked with them on topics of interest.
Focus groups
Focus groups are often condescending. But these are not always older women who are going to discuss a new label for yogurt. This method allows you to quickly gather information, while having a rather large audience. Having conducted several focus groups with players of different ages and preferences, you will understand the general situation by genre, form the segmentation hypotheses, determine their attitude to competitors, general preferences for mechanics, attitude to monetization.
What is important to understand about the focus group? There is what people say (their social role), and there is reality. For example, if you ask a person what he eats, most likely in the wake of the HYIP in a healthy way of life, he will answer that he eats only healthy food, after which the noodles will brew at home.
Players always say that the graphics in the game are very important to them. In this case, they may have very simple laptops that simply do not pull such graphics. But ordinary people often just do not realize it. In surveys about races, we were often told that the racing game should have cool physics and realistic control, but you really can only check whether they want to do this by watching how they play. Ethnography helps us in this.
Ethnography
Interview players at home, in the “habitat” is one of the important elements of the study.
“Ethnography is when you observe people in the environment where they will use your product. They play at home - test at home, navigator for the car - in the car. ”
It is very important to understand where and how your product will be used. In the case of racing games, for example, players who claimed that control in races should be realistic, crashing into a pole, did not leave it, but pressed the quick return button on the track.
“We went to the people at home, where they really will play. For example, it happened that we communicated with children-children and their mothers said: “He will not spend any money on games!” I had to kick them out and talk to the child one-on-one. Then, of course, it turned out quite another. "
Task.
The developer wants to make a mobile version of the game popular desktop genre. How to understand what can be waived and what is not in the conditions of device limitations?
“In order to prioritize the features, we conducted focus groups. We needed to understand why they play the game? What do they like? What is important and what is not for them? This helped us choose the main features in order to port the game to the mobile platform. ”
In addition, competitors were tested to find their strengths and weaknesses.
At this stage it is important to understand how playable is what you came up with. It is necessary to conduct tests on the target groups in order to understand whether the game comes to them or not. Also at this stage you can see what the players themselves understand, and what should be included in the tutorial.
Possible questions
• Do players understand how to play?
• Are there any critical gameplay issues?
• Where is it critical to do the training, and where can they figure it out?
Task.
The first test of the main gameplay mobile CCI. There were several ready-made levels and a rather complicated card pumping system. To identify the main problems and complexities of the gameplay, classical playtests were conducted.
It is necessary to check the understanding of gameplay and game perception by different groups of players. We called them for testing and analyzed who knew how to play and who did not, and, most importantly, who liked it and who did not like it. We obtained quite interesting results. For example, farmers did not understand anything and they did not like it, but the battlers also understood little, but they still liked it.
“A classic UX test is a collection of tasks. A person comes to the laboratory, performs them, and we observe him and, if necessary, ask additional questions. ”
Possible questions
• Do players find their own optimal instance passing strategy?
• Are the monster abilities clear to the players?
• Is the level with ice figures not too difficult?
• Does the video explain the background to the game well?
Task
Check if the players understand the effects and abilities of different mobs and find the strategy to fight against them.
To answer this question, we conducted a playtest with a retrospective. In a special arena, players fought with several packs of mobs. After they looked through their game and told them how they understood the actions of the mobs in each pack and why they chose this or that strategy.
“Each of the mobs can do something: someone lets out a ray, someone makes puddles, someone explodes, and someone calls minions. It was necessary to understand whether the players see these effects, whether they understand the strategies of dealing with them. ”
It turned out that the most noticeable effects are puddles and rays. The rest caused difficulties, for example, people did not notice the countdown of exploding mobs.
“The most visible to a player effect is a puddle, if a player sees a puddle he is running out of it.”
Task.
How to understand if an instance is well made in an MMORPG game?
It was not enough to find UX-problems of gameplay. In addition to the survey of respondents, using psycho-physiological indicators, one can understand whether involvement remains. For this, we conducted a psycho-physiological test. Galvanic skin response is displayed on the graph of monsters in the game. So we were able to compare different packs of monsters on the floors.
Your game is already played, but it would be good to watch how people do it. You can see if the tutorial works. You can test the threshold of entry into the game, see if there are any problems in the first hours of the game.
Possible questions
• Do problems occur during the first hour of play?
• Is training effective?
• Which parts of the game are the most interesting?
• Do players get bored?
Task.
The first hour of the game is critical for a new player. How to reduce the percentage of players leaving at this stage?
The first hour of the game is very important, if a player leaves you immediately, he will not return.
“In collect 3 in a row of the game, we tested how people master the game, at what point they understand control. The game had goals at each level and a limit on the number of moves. For example, collect 10 cherries in 15 turns.
We tracked the players view. Where is their attention concentrated? Initially, while the first levels were going, users only looked at the field. Only after another failure, they began to realize that there were limitations in the game, and to look at the number of moves, on the goals. ”
It was necessary to understand at what point the difficulty appears. We all know that people do not read tutorials. In this case, problems with the objectives of the level and the moves arose, despite the fact that the training told about them. Then watching how people watch tutorials, we can evaluate their effectiveness. We have seen that if some tips worked more or less, but some did not work at all. If the tip highlighted the element and offered to do something in the game, people reluctantly used it, immediately starting to play. Two chances in a row were especially unlikely.
Possible questions
• Why do players fall off after level five?
• Do players understand how to buy in a game store?
• Is the new craft interface effective?
Task.
Compare our game with competitors to identify strengths and weaknesses and understand where to go next.
“We had a case with mobile poker. Everything was good, but the developers wanted to understand where they should develop. We started testing other pokers, watched how people played them, watched where, how and what worked. Thus, we found the most effective features. ”
Task.
To test the new opportunity to change the class in the game. Identify problems with the new interface.
Everything is done here using the classic UX test, so we found interface problems. For example, the absence of warnings, the difficult choice of a new gear, etc.
Task.
Test the process of the first entry into the game: from the landing to download the client. Find game site problems.
With the help of usability testing, we check if the players find the client installation, what they want to see on the site, if they have any problems with registration.
Now, knowing the possible questions for UX testing, we move on to the next item, namely, “How and where should we do a UX test”.
There are several options for doing UX research.
1. External agencies. You can contact your UX testing agency. There are many companies with different specializations. But as outsiders, they are not sufficiently involved in your project. You will have to convey to the researchers in great detail all the details.
2. Organize your laboratory, hire researchers, which is quite expensive, not everyone has the resources for this.
3. On your own, but you need some qualifications to do it well. Conducting research yourself, you risk getting an answer that does not correspond to reality. Especially easy to get the answer that you want.
What the research process consists of:
1. First, we formulate the questions and goals. Why do we need testing.
2. Choose a research method that best meets our goals. (focus group, etc.)
3. Make a plan: a list of questions, tasks, the order of the survey. What people will do and in what order.
4. Choose an audience. Who are we checking for?
5. Conduct research.
6. Analyzing the data.
And here we come to the research methods.
Let's talk about the most popular research methods.
• Is the method suitable for this task?
• Is research needed at all or is there enough statistics?
1) Ethnography
What it is: Observing players in the environment in which they play your game.
Why: Deeply study the audience, understand the context and features of the game behavior, find insights. Understand her desires and problems.
It is good not only to talk with people, but also to watch how they play. In this case, we do it right in their home. Ethnography helps to understand how the process of the game takes place in people’s lives, how it fits into their daily routine. How much do they play? What device? In what conditions? Does something distract them during the game?
The optimal number of respondents is usually not less than 20-25 people.
2) Focus groups
What it is: 6-8 people discuss the questions proposed by the moderator
Why: quickly gather opinions, understand the context, opinions about competitors and genre.
Focus groups - we gather 6-8 people in one place, the moderator asks questions.
Problems: an active person can get caught up with whom everyone will agree, and someone will be silent all the time. Therefore, the role of an experienced moderator who knows how to conduct a conversation is so important.
3) Playtest or usability testing
What it is: You watch people play your game.
Why: Find problems, test hypotheses, collect impressions.
A person comes to you (or you meet on neutral territory) and plays your game, and you observe and draw conclusions. Understand the problems of gameplay, insights about the perception of the game and the behavior of players.
The number of respondents may be quite small. We usually hold playtests for 8-12 people.
Aytrecking
Aitracking is not a separate method, it is a technology within the framework of playtest.
What it is: Fixing the movement of the gaze of respondents during the game.
Why: Find the problems of visibility and focus of attention of respondents. Increase the visibility of research results.
During the eytreking, an analysis of the movement of a person’s gaze takes place: where it looks, and where it doesn’t, whether it sees contextual actions, or looks at the tutorial. You can use eytreking during playtest.
Psychophysiology
Like lighttracking, it is a technology within playtest.
What is it: Fixation of an encephalogram, KGR, ECG and other biometric data
Why: Emotional background and mental load during the game
Sensors are installed on the respondent to analyze its indicators: pulse, galvanic skin response, encephalogram.
4) Diary studies
What it is: Players play at home for a week or two, regularly filling out questionnaires
Why: Check what is impossible in an hour or two in the laboratory. Understand how the interest in the game changes in a day / two / week, and what influences it. Understand how players master the game, what strategies they choose.
An hour in the laboratory is not always possible to check your game, but there is a way out. People, playing at home, are constantly in touch with you, they may carry out your tasks, they answer in the questionnaire prepared by you. So you can check the involvement in the game, find problems.
And it also happens: a person plays a game at home during the week, and then comes to the laboratory for research already prepared.
The optimal number of respondents: from 15 people. But remember that part of the respondents may “fall off”, not having sustained the full term of the study. Usually such percentages are 15-20.
Respondents
When the goal is set, you have decided on the method, the question arises, who to test, who to call for research? You must clearly define the target audience of your game. Is this person already playing your game or has he never seen her before? Does he play similar games? What is his experience? Who is he: casual, midkorschik or hardcore?
“When we ask to tell who are casual?” All respond differently. There is no clear wording. It all depends on your understanding of the issue.
What games does a person play? How many plays? What is the length of the session? For each game, the answer may be different. ”
Decided on the target audience? Great, now you need to decide where to look for these people.
1. Relatives and colleagues
Pros: ready to help for free.
Cons: most often they are not your c / a.
People who are loyal to you will either treat everything too softly, or vice versa - too critical.
2. Agencies and panels
Pros: there is no headache with the search
Cons: need a budget, there are "professional walkers"
The agency specially trained people will find you for the money people meet your requirements. This is convenient, but it is likely that among these people there will be professional respondents who are ready to dishonestly answer your questions for money. This will spoil the test results. There is a rule that a person should go for research no more than once every six months.
We need to come up with methods, how to identify people who lie. For example, check on a person’s account how many hours are played.
3. Forum and game publics
Pros: Loyal audience, ready to help. Often free or for a gift.
Cons:: Only a hardcore audience, there are freaks, excessive demands on the game. They take the test as an opportunity to once again ask for something that is already being asked on the forum.
Forum or game public is a good source of the audience that will come to your aid. This is a loyal audience to you. But the forum does not help to find people who are not playing your game or casual audience.
4. Other options
You can try to find people on:
• Game publics and resources
• In computer clubs
• Among the audience of other games
• On YouDo, WorkZilla - but there, too, need to be careful and check people
Single or collective?
Who will the respondent play with? Bots, online players, your colleagues or other testers?
In a group test, moderate the conversations of players in the process so that the opinion of one player does not affect the opinion of his neighbor.
You need to clearly understand whether the person will play alone, in a group, with a bot, or, perhaps, with your developer, who will be sitting at the wall.
During the group test you need to monitor whether the players communicate with each other during the test. If they start to discuss something during the game, it may affect their opinion, thereby blurring the overall picture. All communication between respondents with each other must be moderated.
Do not forget about the natural context. Try to create conditions for the game as close to life as possible. If this is a mobile game, think about how the person will hold the device, whether there will be distractions, how stable wi-fi should be.
For example, if you need to play on a tablet, the easiest way is to put it on the table, but there is no natural context, so you can conduct several tests: with a tablet on the table and with a tablet in your hands / on your lap. That is, it is important to balance between the natural and the simplest options.
In order to independently conduct a playtest, you need:
1. Separate room or conversation. It is important that a person is not distracted. Before testing, you need to talk to a person so that he relaxes and does not feel experimental.
2. Device for the test. It is advisable to use the respondent’s personal device.
3. Organize the broadcast. It is desirable that the development team watched the broadcast in another room, but in any case not at the shoulder of the player.
3. It will be good if you can record testing on video. What is happening on the screen, and directly what is happening in the room, the respondent himself.
Screen capture and webcam software: Morae, Camtasia, Camstudio.
First, you need a script (plan). You need to think about what elements the respondent will play, what to show him. In this regard, with games it is a little easier as the game itself leads it according to the scenario. A list of tasks and questions for the respondent is also required. For example, a person must pass the first three levels, then we artificially pump it over and allow four more to pass.
“It is very important to formulate the task correctly. If you tell the person that he has a goal in the instance to kill the boss, he can just simply run around all the other mobs. ”
Stay as neutral as possible, do not ask leading questions. Wrong question or hint can greatly affect the results of the study. If the test is conducted by the person who developed the game, he always has some expectations that affect the passing of the test. Therefore, he can be a bad moderator, his prejudices will inevitably affect the test, the temptation to give a hint is very high. Forget all your prejudices and test as if they were not there.
Sometimes it is useful to conduct external research because external researchers have no bias towards your product. Also, if you know about a problem in your game, you cannot purposefully draw the respondent’s attention to this problem, you can only create the conditions in which he could face it.
“ , , . , : “ ””.
No one is able to adequately answer the question “Will it be cool if we do this?” Or “What can we do to make it better?”. A person very poorly models his future behavior.
Players can only answer about past experience. Therefore, it is better to ask what the player likes in games where a similar feature is implemented, whether he uses it, etc.
Do not ask respondents closed questions. You can’t ask “Did you like it?” The answer will be “Yes!” Or “No”, and most likely “Yes!” Due to social desirability. It is better to ask “What did you like? What didn't you like? ”
“ . , . , , - , . , , . , . , , , . , ”.
No need to extrapolate statistical findings from qualitative research to the entire audience. Do not confuse qualitative and quantitative research. In qualitative research, one must pay attention to the problems themselves, and not to the number of people who have found them. Look for insights, but do not collect statistics if you have a small sample.
Individual behavior can also be valuable, but do not mislead anyone.
“Even if only one person found the problem, it can be valuable.”
Research is often stressful for a designer. Be prepared for the fact that people can say unpleasant things and scold your child. Do not argue with the players, even if they in vain scold your game. Find out the reasons for their discontent and draw conclusions.
If you are not sure that the problem found may appear again, do some more tests.
Games are different from other products in that in addition to the convenience they have a fan. A separate task for testing is to understand how good your players were. It is not always possible to determine this in the process of observing them, and in interviews they often note both good and bad. Here questionnaires will help you, so players will evaluate the game on a single pattern for all. There are several ready-made questionnaires that can be used.
MICROSOFT DESIRABILITY TOOLKIT
Questionnaire consisting of positive and negative adjectives. Quick assessment of the associations and characteristics of the game.
Full description.
GAME EXPERIENCE QUESTIONNAIRE
The universal game questionnaire created by Jeanne H. Brockmyer from the University of Toledo helps to evaluate the game according to basic parameters (game world, immersion in the game, tension, motivation to act, etc.).
Full description.
The main thing, what conclusions do you draw from the results of the study. You can do a cool study, but not do anything good for the product.
What should I do to make the research truly useful?
For everything to work, you must be involved. The most important thing is for you to watch, participate and feel it. You need to understand whether your management and colleagues believe in the study. If not, then you need to call them and show them the videotapes, try to get everyone involved to the maximum, this will increase the degree of trust in the results, and therefore the likelihood that something will be done.
“Team faith is very important. This is a common mistake of outside researchers. It seems to them that they should just provide a research report, but the main task of the researcher is to make the customers believe in the results. ”
The task of the researcher is to make the results go into your head, and not just provide you with a beautiful report. The research itself will not make your game better, you must do it.
Video from the master class "Custom Studies in Games", Natalia Sprogis:
We will be happy to discuss your questions in the comments. If you want to attend our live gaming industry workshops, then come on February 11 for the lecture day , where from 12:00 to 21:00 there will be lectures on game design and the basics of game development from the GBVI teachers and invited experts from the game industry. And tomorrow, by the way, we will also have an open day .
In this article we will answer the questions that were discussed at the master class:
- Why do game developers need UX research?
- What questions do they help answer?
- At what point in the development should they be used?
- How to organize a study?
')
Many experts from the game industry believe that research is not needed, or only big companies need it. Usually they give the following arguments:
- Why do we need research: we have experts who understand what to do and how.
- Experts are very good, but all that they have invented should be checked on living people. Why?
- Because you can come up with a great game, but present it to the wrong audience;
- Because you can come up with an excellent game, but to embody it in such a way that your audience will not understand it.
We have testers, so we don’t need research.
Testers check how well the game fits your idea, but they can't test how good your idea is.
We have beta testing with players, why do we need research?
First, the beta tests focus on the technical component.
Secondly, the most active and advanced players participate in beta. You get feedback and bug reports from activists and hardcore players, but you still don’t know why a regular player left you.
Why do everyone need UX research?
1. They help to understand a lot about your game. For example:
- Why do players leave after the first hour of the game?
- Why is the new quest a very low percentage of passage?
- Why is your game not like casual?
- Why do you play only non-paying users?
“Let's say you fall off part of the players after the 5th level. You need to understand why this is happening. Perhaps they are infuriated by the boss or the level is too difficult or boring. The user is tired and uninteresting to him. ”
2. They help to see the problems of gameplay and usability. During the study, you will be able to trace the user's behavior and use this knowledge in the future to correct errors.
3. They help you get to know your audience. Perhaps you imagine it completely differently. It happens that the development team is shocked when it meets their players for the first time on a test.
When should research be applied? There are different types of research suitable for each stage of development. They all answer different questions.
Stage 1. Concepting and prototyping
Possible issues at this stage of development:
• At what age of players should we be better oriented?
• What do players like about competitors? And what mistakes do not need to be repeated?
• Do we have chances to hook casual players?
• What features should be developed in the first place, and what can wait?
Consider a few examples of research at the stage of prototyping.
Case 1. Segmentation
There is a certain game genre. The developer wants to make a game in this genre, but he does not have a clear vision of what it should be. Our task is to conduct research in order to understand what the players love in this genre, as well as to highlight groups of players according to their behavior and expectations from the game. To do this, we first conducted six focus groups. We collected 6-8 people of a certain profile and talked with them on topics of interest.
Focus groups
Focus groups are often condescending. But these are not always older women who are going to discuss a new label for yogurt. This method allows you to quickly gather information, while having a rather large audience. Having conducted several focus groups with players of different ages and preferences, you will understand the general situation by genre, form the segmentation hypotheses, determine their attitude to competitors, general preferences for mechanics, attitude to monetization.
What is important to understand about the focus group? There is what people say (their social role), and there is reality. For example, if you ask a person what he eats, most likely in the wake of the HYIP in a healthy way of life, he will answer that he eats only healthy food, after which the noodles will brew at home.
Players always say that the graphics in the game are very important to them. In this case, they may have very simple laptops that simply do not pull such graphics. But ordinary people often just do not realize it. In surveys about races, we were often told that the racing game should have cool physics and realistic control, but you really can only check whether they want to do this by watching how they play. Ethnography helps us in this.
Ethnography
Interview players at home, in the “habitat” is one of the important elements of the study.
“Ethnography is when you observe people in the environment where they will use your product. They play at home - test at home, navigator for the car - in the car. ”
It is very important to understand where and how your product will be used. In the case of racing games, for example, players who claimed that control in races should be realistic, crashing into a pole, did not leave it, but pressed the quick return button on the track.
“We went to the people at home, where they really will play. For example, it happened that we communicated with children-children and their mothers said: “He will not spend any money on games!” I had to kick them out and talk to the child one-on-one. Then, of course, it turned out quite another. "
Case 2. Prioritize features
Task.
The developer wants to make a mobile version of the game popular desktop genre. How to understand what can be waived and what is not in the conditions of device limitations?
“In order to prioritize the features, we conducted focus groups. We needed to understand why they play the game? What do they like? What is important and what is not for them? This helped us choose the main features in order to port the game to the mobile platform. ”
In addition, competitors were tested to find their strengths and weaknesses.
Stage 2. Vertical cut
At this stage it is important to understand how playable is what you came up with. It is necessary to conduct tests on the target groups in order to understand whether the game comes to them or not. Also at this stage you can see what the players themselves understand, and what should be included in the tutorial.
Possible questions
• Do players understand how to play?
• Are there any critical gameplay issues?
• Where is it critical to do the training, and where can they figure it out?
Case 3. Test CORE GAMEPLAY
Task.
The first test of the main gameplay mobile CCI. There were several ready-made levels and a rather complicated card pumping system. To identify the main problems and complexities of the gameplay, classical playtests were conducted.
It is necessary to check the understanding of gameplay and game perception by different groups of players. We called them for testing and analyzed who knew how to play and who did not, and, most importantly, who liked it and who did not like it. We obtained quite interesting results. For example, farmers did not understand anything and they did not like it, but the battlers also understood little, but they still liked it.
“A classic UX test is a collection of tasks. A person comes to the laboratory, performs them, and we observe him and, if necessary, ask additional questions. ”
Stage 3. Content production
Possible questions
• Do players find their own optimal instance passing strategy?
• Are the monster abilities clear to the players?
• Is the level with ice figures not too difficult?
• Does the video explain the background to the game well?
Case 4. Test armies mobs
Task
Check if the players understand the effects and abilities of different mobs and find the strategy to fight against them.
To answer this question, we conducted a playtest with a retrospective. In a special arena, players fought with several packs of mobs. After they looked through their game and told them how they understood the actions of the mobs in each pack and why they chose this or that strategy.
“Each of the mobs can do something: someone lets out a ray, someone makes puddles, someone explodes, and someone calls minions. It was necessary to understand whether the players see these effects, whether they understand the strategies of dealing with them. ”
It turned out that the most noticeable effects are puddles and rays. The rest caused difficulties, for example, people did not notice the countdown of exploding mobs.
“The most visible to a player effect is a puddle, if a player sees a puddle he is running out of it.”
Case 5. Test gaming instance
Task.
How to understand if an instance is well made in an MMORPG game?
It was not enough to find UX-problems of gameplay. In addition to the survey of respondents, using psycho-physiological indicators, one can understand whether involvement remains. For this, we conducted a psycho-physiological test. Galvanic skin response is displayed on the graph of monsters in the game. So we were able to compare different packs of monsters on the floors.
Stage 4. PTA and MBT
Your game is already played, but it would be good to watch how people do it. You can see if the tutorial works. You can test the threshold of entry into the game, see if there are any problems in the first hours of the game.
Possible questions
• Do problems occur during the first hour of play?
• Is training effective?
• Which parts of the game are the most interesting?
• Do players get bored?
Case 6. Lower entry threshold
Task.
The first hour of the game is critical for a new player. How to reduce the percentage of players leaving at this stage?
The first hour of the game is very important, if a player leaves you immediately, he will not return.
“In collect 3 in a row of the game, we tested how people master the game, at what point they understand control. The game had goals at each level and a limit on the number of moves. For example, collect 10 cherries in 15 turns.
We tracked the players view. Where is their attention concentrated? Initially, while the first levels were going, users only looked at the field. Only after another failure, they began to realize that there were limitations in the game, and to look at the number of moves, on the goals. ”
It was necessary to understand at what point the difficulty appears. We all know that people do not read tutorials. In this case, problems with the objectives of the level and the moves arose, despite the fact that the training told about them. Then watching how people watch tutorials, we can evaluate their effectiveness. We have seen that if some tips worked more or less, but some did not work at all. If the tip highlighted the element and offered to do something in the game, people reluctantly used it, immediately starting to play. Two chances in a row were especially unlikely.
Stage 5. After release
Possible questions
• Why do players fall off after level five?
• Do players understand how to buy in a game store?
• Is the new craft interface effective?
Case 7. Comparison with competitors
Task.
Compare our game with competitors to identify strengths and weaknesses and understand where to go next.
“We had a case with mobile poker. Everything was good, but the developers wanted to understand where they should develop. We started testing other pokers, watched how people played them, watched where, how and what worked. Thus, we found the most effective features. ”
Case 8. Class change in MMORPG
Task.
To test the new opportunity to change the class in the game. Identify problems with the new interface.
Everything is done here using the classic UX test, so we found interface problems. For example, the absence of warnings, the difficult choice of a new gear, etc.
Case 9. Related Media
Task.
Test the process of the first entry into the game: from the landing to download the client. Find game site problems.
With the help of usability testing, we check if the players find the client installation, what they want to see on the site, if they have any problems with registration.
Now, knowing the possible questions for UX testing, we move on to the next item, namely, “How and where should we do a UX test”.
How to organize all this?
There are several options for doing UX research.
1. External agencies. You can contact your UX testing agency. There are many companies with different specializations. But as outsiders, they are not sufficiently involved in your project. You will have to convey to the researchers in great detail all the details.
2. Organize your laboratory, hire researchers, which is quite expensive, not everyone has the resources for this.
3. On your own, but you need some qualifications to do it well. Conducting research yourself, you risk getting an answer that does not correspond to reality. Especially easy to get the answer that you want.
What the research process consists of:
1. First, we formulate the questions and goals. Why do we need testing.
2. Choose a research method that best meets our goals. (focus group, etc.)
3. Make a plan: a list of questions, tasks, the order of the survey. What people will do and in what order.
4. Choose an audience. Who are we checking for?
5. Conduct research.
6. Analyzing the data.
And here we come to the research methods.
Methods
Let's talk about the most popular research methods.
• Is the method suitable for this task?
• Is research needed at all or is there enough statistics?
1) Ethnography
What it is: Observing players in the environment in which they play your game.
Why: Deeply study the audience, understand the context and features of the game behavior, find insights. Understand her desires and problems.
It is good not only to talk with people, but also to watch how they play. In this case, we do it right in their home. Ethnography helps to understand how the process of the game takes place in people’s lives, how it fits into their daily routine. How much do they play? What device? In what conditions? Does something distract them during the game?
The optimal number of respondents is usually not less than 20-25 people.
2) Focus groups
What it is: 6-8 people discuss the questions proposed by the moderator
Why: quickly gather opinions, understand the context, opinions about competitors and genre.
Focus groups - we gather 6-8 people in one place, the moderator asks questions.
Problems: an active person can get caught up with whom everyone will agree, and someone will be silent all the time. Therefore, the role of an experienced moderator who knows how to conduct a conversation is so important.
3) Playtest or usability testing
What it is: You watch people play your game.
Why: Find problems, test hypotheses, collect impressions.
A person comes to you (or you meet on neutral territory) and plays your game, and you observe and draw conclusions. Understand the problems of gameplay, insights about the perception of the game and the behavior of players.
The number of respondents may be quite small. We usually hold playtests for 8-12 people.
Aytrecking
Aitracking is not a separate method, it is a technology within the framework of playtest.
What it is: Fixing the movement of the gaze of respondents during the game.
Why: Find the problems of visibility and focus of attention of respondents. Increase the visibility of research results.
During the eytreking, an analysis of the movement of a person’s gaze takes place: where it looks, and where it doesn’t, whether it sees contextual actions, or looks at the tutorial. You can use eytreking during playtest.
Psychophysiology
Like lighttracking, it is a technology within playtest.
What is it: Fixation of an encephalogram, KGR, ECG and other biometric data
Why: Emotional background and mental load during the game
Sensors are installed on the respondent to analyze its indicators: pulse, galvanic skin response, encephalogram.
4) Diary studies
What it is: Players play at home for a week or two, regularly filling out questionnaires
Why: Check what is impossible in an hour or two in the laboratory. Understand how the interest in the game changes in a day / two / week, and what influences it. Understand how players master the game, what strategies they choose.
An hour in the laboratory is not always possible to check your game, but there is a way out. People, playing at home, are constantly in touch with you, they may carry out your tasks, they answer in the questionnaire prepared by you. So you can check the involvement in the game, find problems.
And it also happens: a person plays a game at home during the week, and then comes to the laboratory for research already prepared.
The optimal number of respondents: from 15 people. But remember that part of the respondents may “fall off”, not having sustained the full term of the study. Usually such percentages are 15-20.
Respondents
When the goal is set, you have decided on the method, the question arises, who to test, who to call for research? You must clearly define the target audience of your game. Is this person already playing your game or has he never seen her before? Does he play similar games? What is his experience? Who is he: casual, midkorschik or hardcore?
“When we ask to tell who are casual?” All respond differently. There is no clear wording. It all depends on your understanding of the issue.
What games does a person play? How many plays? What is the length of the session? For each game, the answer may be different. ”
Decided on the target audience? Great, now you need to decide where to look for these people.
1. Relatives and colleagues
Pros: ready to help for free.
Cons: most often they are not your c / a.
People who are loyal to you will either treat everything too softly, or vice versa - too critical.
2. Agencies and panels
Pros: there is no headache with the search
Cons: need a budget, there are "professional walkers"
The agency specially trained people will find you for the money people meet your requirements. This is convenient, but it is likely that among these people there will be professional respondents who are ready to dishonestly answer your questions for money. This will spoil the test results. There is a rule that a person should go for research no more than once every six months.
We need to come up with methods, how to identify people who lie. For example, check on a person’s account how many hours are played.
3. Forum and game publics
Pros: Loyal audience, ready to help. Often free or for a gift.
Cons:: Only a hardcore audience, there are freaks, excessive demands on the game. They take the test as an opportunity to once again ask for something that is already being asked on the forum.
Forum or game public is a good source of the audience that will come to your aid. This is a loyal audience to you. But the forum does not help to find people who are not playing your game or casual audience.
4. Other options
You can try to find people on:
• Game publics and resources
• In computer clubs
• Among the audience of other games
• On YouDo, WorkZilla - but there, too, need to be careful and check people
Testing organization
Single or collective?
Who will the respondent play with? Bots, online players, your colleagues or other testers?
In a group test, moderate the conversations of players in the process so that the opinion of one player does not affect the opinion of his neighbor.
You need to clearly understand whether the person will play alone, in a group, with a bot, or, perhaps, with your developer, who will be sitting at the wall.
During the group test you need to monitor whether the players communicate with each other during the test. If they start to discuss something during the game, it may affect their opinion, thereby blurring the overall picture. All communication between respondents with each other must be moderated.
Do not forget about the natural context. Try to create conditions for the game as close to life as possible. If this is a mobile game, think about how the person will hold the device, whether there will be distractions, how stable wi-fi should be.
For example, if you need to play on a tablet, the easiest way is to put it on the table, but there is no natural context, so you can conduct several tests: with a tablet on the table and with a tablet in your hands / on your lap. That is, it is important to balance between the natural and the simplest options.
In order to independently conduct a playtest, you need:
1. Separate room or conversation. It is important that a person is not distracted. Before testing, you need to talk to a person so that he relaxes and does not feel experimental.
2. Device for the test. It is advisable to use the respondent’s personal device.
3. Organize the broadcast. It is desirable that the development team watched the broadcast in another room, but in any case not at the shoulder of the player.
3. It will be good if you can record testing on video. What is happening on the screen, and directly what is happening in the room, the respondent himself.
Screen capture and webcam software: Morae, Camtasia, Camstudio.
Holding
First, you need a script (plan). You need to think about what elements the respondent will play, what to show him. In this regard, with games it is a little easier as the game itself leads it according to the scenario. A list of tasks and questions for the respondent is also required. For example, a person must pass the first three levels, then we artificially pump it over and allow four more to pass.
“It is very important to formulate the task correctly. If you tell the person that he has a goal in the instance to kill the boss, he can just simply run around all the other mobs. ”
Stay as neutral as possible, do not ask leading questions. Wrong question or hint can greatly affect the results of the study. If the test is conducted by the person who developed the game, he always has some expectations that affect the passing of the test. Therefore, he can be a bad moderator, his prejudices will inevitably affect the test, the temptation to give a hint is very high. Forget all your prejudices and test as if they were not there.
Sometimes it is useful to conduct external research because external researchers have no bias towards your product. Also, if you know about a problem in your game, you cannot purposefully draw the respondent’s attention to this problem, you can only create the conditions in which he could face it.
“ , , . , : “ ””.
No one is able to adequately answer the question “Will it be cool if we do this?” Or “What can we do to make it better?”. A person very poorly models his future behavior.
Players can only answer about past experience. Therefore, it is better to ask what the player likes in games where a similar feature is implemented, whether he uses it, etc.
Do not ask respondents closed questions. You can’t ask “Did you like it?” The answer will be “Yes!” Or “No”, and most likely “Yes!” Due to social desirability. It is better to ask “What did you like? What didn't you like? ”
Results analysis
“ . , . , , - , . , , . , . , , , . , ”.
Statistics
No need to extrapolate statistical findings from qualitative research to the entire audience. Do not confuse qualitative and quantitative research. In qualitative research, one must pay attention to the problems themselves, and not to the number of people who have found them. Look for insights, but do not collect statistics if you have a small sample.
Individual behavior can also be valuable, but do not mislead anyone.
“Even if only one person found the problem, it can be valuable.”
Research is often stressful for a designer. Be prepared for the fact that people can say unpleasant things and scold your child. Do not argue with the players, even if they in vain scold your game. Find out the reasons for their discontent and draw conclusions.
If you are not sure that the problem found may appear again, do some more tests.
How will the fan measure?
Games are different from other products in that in addition to the convenience they have a fan. A separate task for testing is to understand how good your players were. It is not always possible to determine this in the process of observing them, and in interviews they often note both good and bad. Here questionnaires will help you, so players will evaluate the game on a single pattern for all. There are several ready-made questionnaires that can be used.
MICROSOFT DESIRABILITY TOOLKIT
Questionnaire consisting of positive and negative adjectives. Quick assessment of the associations and characteristics of the game.
Full description.
GAME EXPERIENCE QUESTIONNAIRE
The universal game questionnaire created by Jeanne H. Brockmyer from the University of Toledo helps to evaluate the game according to basic parameters (game world, immersion in the game, tension, motivation to act, etc.).
Full description.
What if it doesn't work?
The main thing, what conclusions do you draw from the results of the study. You can do a cool study, but not do anything good for the product.
What should I do to make the research truly useful?
For everything to work, you must be involved. The most important thing is for you to watch, participate and feel it. You need to understand whether your management and colleagues believe in the study. If not, then you need to call them and show them the videotapes, try to get everyone involved to the maximum, this will increase the degree of trust in the results, and therefore the likelihood that something will be done.
“Team faith is very important. This is a common mistake of outside researchers. It seems to them that they should just provide a research report, but the main task of the researcher is to make the customers believe in the results. ”
The task of the researcher is to make the results go into your head, and not just provide you with a beautiful report. The research itself will not make your game better, you must do it.
Video from the master class "Custom Studies in Games", Natalia Sprogis:
Part 1
Part 2
Part 3
We will be happy to discuss your questions in the comments. If you want to attend our live gaming industry workshops, then come on February 11 for the lecture day , where from 12:00 to 21:00 there will be lectures on game design and the basics of game development from the GBVI teachers and invited experts from the game industry. And tomorrow, by the way, we will also have an open day .
Source: https://habr.com/ru/post/319890/
All Articles