Hi, Habr! This is Natalia Sprogis, Head of UX-research at Mail.Ru Group. I continue the cycle of articles about the intricacies of the organization and research. Today we will talk about how a usability testing moderator should behave. A moderator is a person who interacts with the respondent: he meets him, instructs and introduces an interview, sets up equipment for recording video, suggests text tasks and asks clarifying and final questions. It would seem that if there is a good scenario (how to compose it, you can read in the planning of usability testing. Part 2 ), it is enough to follow it clearly and not confuse anything. However, the moderator may, voluntarily or unwittingly, greatly influence the respondent, and as a result, the test results. Therefore, it is important to understand how to behave on the test, when and what questions to ask and how to deal with complex respondents.

The commandments of a good moderator

"Shut up and watch." The usability testing moderator has two important tasks:

1. Ensure that the respondent performs testing under conditions closest to natural.
2. Get maximum information about the reasons for the respondent’s actions and his emotions caused by working with the product.

Often, moderators fixate on the second task, bombarding the user with endless “why?”, “Why?”, “How?”, Etc. Many articles on moderation are about how to ask questions correctly. This is an important skill, and we'll definitely talk about it. But in fact, the main thing for a moderator is to learn how to remain silent . Indeed, in a real situation, the respondent will use the product independently. By asking too many questions, you shift a person’s focus from using the product to discussing it. And the most valuable observations of the activities of the respondent. The question can lead him astray and distract from the task. Perhaps, returning to the task, the testing participant will begin to act differently. Therefore, ideally ask questions after the task (in some cases - after the test). Or at least wait until the respondent, completing the task, reaches the logical point.
')
"Do no harm . " The medical commandment is also relevant in usability testing. It is very easy to question, test or imply bias with a question, word or even a gesture. And most importantly, this error may not be visible to any moderator or observers. The moderator during the test should remain absolutely neutral, and it is not so simple. Motivate the respondent to say more, but do not influence what he says. I was faced with a situation where a moderator, confident that there was a problem in a certain place of the project, I tried to find it with every respondent, even those who had not encountered it. Concluded the discussion on the illogicality of this moment, pushing users to recognize: yes, there is a problem. The edge is pretty thin. If you have a hypothesis, be sure to test it on all respondents, giving them relevant tasks or asking questions for understanding. But if the respondent did not encounter a problem in his work, pointing to it and discussing it introduces bias.

Respondents can even read non-verbal signs. For example, if you are well prepared for testing, the script probably contains a set of hypotheses. In some places you foresee and anticipate problems. Or you can empathize with the project team and be loyal to the product being researched. And people love to behave socially desirable. Test participants may feel that you feel uncomfortable when they scold a project. Or that you are inspired when they talk about a particular problem.
Watch not only for words, but also for gestures, posture. Even taking notes during the test is best done all the time, and not just at interesting moments, so as not to show the respondent what his words or actions are really important. Excess then simply do not consider.

"Do not answer questions . " Very often you will be faced with questions from respondents. For example, “Do I need to register now?”, “Am I acting correctly?”, “What should I do now?” . You do not need to answer them. Otherwise, the respondent will also rely on you in the next difficult situation and will not try to handle it himself. It is better to react like this:

• “What do you think yourself?”
• "Let's try to figure it out."
• “What would you do in real life if I were not around?”

Sometimes, if the questions are caused by a not quite standard behavior of the system, I even pretend that I myself do not understand what is happening in order to motivate the respondent to take the search into his own hands. But this does not mean that you do not need to help a person who simply does not know how to carry out the task you have compiled.

"Do not explain the operation of the system . " Sometimes, especially if a failure occurred, the moderator seeks to calm the respondent and explain how and why the system behaved differently. But if you explain the behavior of the system to the participant, you can influence the image of the product in his head or tritely give a hint, impossible in a real, not a test situation.

"Do not apologize for the product . " The respondent suffers from the interface. Moderator is watching. He has a desire to apologize for the fact that designers have not thought of something. Do not do this: you will lose neutrality and become a “product advocate”. And it will be harder for the user to give you new comments.

“Show must go on!” No matter how well prepared you are for the test, there is always the possibility that
something goes wrong. The tracker will stop working correctly, programmers will not have time to bring the system to a working state by the test time, a fire alarm will start in the building (and this has happened in my practice), etc. Therefore, be prepared to use plan B. The tracker does not work - we will record a video without him. There is no time for the test to be an important customer - we will adjust the broadcast via the instant messenger to it. The respondent does not install the application on the phone - we will give him our test (although this may be a reason to change the participant, if it is critical that the device is familiar). In addition, provide the respondent with comfortable conditions to wait if the beginning of the test has to be postponed. Offer him refresh tea or coffee. Tell us how to enable guest Wi-Fi, if possible. In our laboratory for such situations is a stack of magazines. In the case of a delay, set yourself a deadline to solve the problem, after which it is better to release the respondent with a part of the remuneration and do not continue to waste time. And most importantly, do not make tragedy out of it. Force majeure happen, which is perfectly normal. In a pinch, the test can simply be transferred.

"Watch for emotions . " This skill comes with time, but at first it is very difficult not to react emotionally to what is happening on the test. The actions and words of the respondent may seem ridiculous to you. Sometimes there are frankly unpleasant and annoying people. Whatever emotions a test participant may cause you, always remain detachedly benevolent. Also follow the good sound insulation. Bursts of laughter and heated discussions in the observant are permissible only if they are not heard by the respondents.

"Make contact . " The point seems obvious. After all, during any work with people, we must find a common language with them. However, often, especially when it is not the first day of testing and not the first respondent for today, the moderator, without noticing it, turns into a car. And we begin to mumble the learned greeting text and try to get rid of the respondent as soon as possible. All this is very visible and felt, and most importantly, it can affect how the respondent will work with the product, whether he will be happy to comment on his actions or, just like you, will approach the test formally. Smile to the person, look into his eyes and “live” the greeting text. Adjust the words, speed, tone under the interlocutor. Even if you leave the respondent alone for the rest of the testing, try to form positive expectations from the event itself (not the product!).

"Be prepared to say goodbye to the wrong person . " Unfortunately, you probably sooner or later encounter the fact that the respondent does not suit you. The reason may be an annoying mistake of recruiting, misunderstanding, and sometimes a banal lie for the sake of reward. We were visited by people who did not use the necessary products, but registered in them yesterday. Periodically bring someone else's phone with the right set of applications. Be prepared to say goodbye to the person if you realize that he does not meet the requirements of the test. Usually, after feeling “something is not right,” we collectively decide with the project team whether this test will give us something. The main thing - make a decision at the very beginning. So you do not spend the time - neither your own nor the respondent. Therefore, I advise you to check the recruiting during the introductory interview.

Moderator falls is not the most pleasant role. If you decide to stop testing, remember: the way you say goodbye to the applicant is also important. Try to make sure that he does not have unpleasant impressions. After all, you are for him - the face of the company. In addition, a person may not be to blame for the error: perhaps it is the recruiter’s fault. A potential respondent spent time and came to you. In most cases, even if I suspect a person in a lie, I say a similar phrase: “Unfortunately, in our conversation, I see that your profile does not quite match the goals of our testing. Therefore, I have to finish it . ” In some cases (especially if you started testing), it is reasonable to give the respondent a reward, in part or even entirely.

Be especially careful with loyal users of the product. They are usually found through communities and project forums, and the motivation to come from them is not financial. The mistake in recruiting here was made, most likely, by the one who found the person. These people are encouraged by the opportunity to participate in the fate of their favorite project. If you see that the applicant is still not quite consistent with the goals of testing, try not to upset him. Better to do a mini-test with him, give him a little use of the product. Let him feel that he came for good reason and was also able to “join” in the development of his favorite project.

How to ask questions

As mentioned above, in contrast to in-depth interviews based on a conversation with the respondent, the main task of usability testing is to monitor the activity . Therefore (in addition to the introductory and final interview) in the main part of the testing, the moderator gives the respondent more opportunity to act, motivating him to share his impressions. However, in order to penetrate the behavior of a person, to clarify his incomprehensible comments, to learn the attitude to certain things, the moderator will have to ask questions. And their role cannot be underestimated. What words, with what intonation and when exactly you will ask a question - all this can influence the behavior of the respondent. So, and on the final test results.

When to ask questions

Questions during the main part of testing are most often needed if the following is required:

• Clarification : clarify / add a respondent’s comment. You ask either to clarify the idea, or to reveal it more fully. For example, you hear: "This is incomprehensible." Specify: “What exactly is not clear? What behavior did you expect? ”
• Explanation : find out the reasons for the behavior of the respondent. For example, you see that he sent a gift to a friend in Odnoklassniki and immediately goes to a friend’s page. Find out why. Perhaps the respondent checks whether a gift has appeared there and did not understand that this happens only after the approval of the recipient.
• Attitude : find out the attitude to the event / behavior of the system or clarify your evaluation comment. Once our respondent described the plot of the game as: “classic light addiction”. The moderator immediately had to clarify: “Is light drug addiction a good or bad thing?” (Thanks to Ekaterina Akatova for an example).
• Background : to assemble the basis for compiling assignments in a flexible scenario or to collect the context for the problem described by the respondent. If you use tasks from the respondent’s experience in testing, then check with the participant for details before each task. For example: “Remember the last time you bought tickets. Where were you flying? What criteria used to select tickets? What about restrictions (transfer, cost, flight time, airline)? And now let's find the air ticket according to the same criteria . ” Sometimes comments about the background are needed to get the real need out of the respondent. For example, a person complains that there is a lack of some kind of functional. Ask him to give examples from life when he needed a similar set of functions.

So, the essence and wording of the question are important. And it is important to set it in time. As I already said in the commandment “Shut up and watch”, if you ask something from the respondent who is performing the task, you will trite him with a thought. The conversation may affect the respondent, which is undesirable. In addition, if the user receives too many questions from the moderator when working with the product, there is a chance that he will switch to the discussion of the product and stop doing something. Therefore, try to save questions before completing the task or at least its block. Make yourself a mark: so you will not forget what caused your interest. Clarifying questions will not interfere if the user himself comments on something or thinks out loud. In this case, you simply motivate the respondent to express his thought in a little more detail. But with this, be careful: if a detailed discussion begins, it will distract from the current task.

What are the questions

Open and closed questions . The first moderator's rule is to ask open-ended questions, i.e., requiring a detailed answer. Your favorite questions should be “why?”, “Why?”, “What do you think about this?” , Etc. In contrast to them, closed questions imply a definite answer (“yes” or “no”). They are bad because users tend to agree more, to answer "yes." This comes from not wanting to argue with the moderator and hurt his feelings. In addition, closed questions can not bring new knowledge: everything that can be learned is inherent in the question itself.

There is nothing worse than asking the respondent after the assignment: “Was it convenient for you?” With a high degree of probability, you will receive mostly affirmative answers no matter how it all happened. If an assessment of an event or a system’s behavior is needed, it’s permissible to ask: “Is this good or bad? Simple or hard? Fast or slow? " So you set the desired rating scale, while remaining neutral.

Preconceived questions You should not be interested after the test: “How do you feel about the fact that the new design is more modern and technological?” It seems to be an open question. But in its formulation it is laid down that the user should perceive the design as modern and technological. And this may not coincide with his opinion. Try to avoid ratings made for the respondent. How to find out whether the user considers the new version more technologically or more modern? There are different ways. To begin with, ask him to describe the version himself with several adjectives that come to mind. Then you can give a list of adjectives (positive and negative) containing the characteristics you expect, and see if the respondent chooses them. A set of adjectives can be obtained from the Microsoft Desirability Toolkit .

Leading questions . Some questions contain a hint. For example, the question "Do you think it is possible to do this on the website?" Is essentially suggestive. After all, it is obvious: why ask about what cannot be done? Sometimes a question may lead the user to think about a product that has not yet occurred to him. Take as an example the chain of letters (threads) in Mail.Ru Mail. During testing, we knew that many users had never encountered this. Questions like “And from whom is this letter below? And on what basis are they grouped? ” Would lead the user to think about how it all works. Therefore, at first we gave the respondents their own ideas on the principle of work and asked only general questions: “What is this? How it works? What happened to your letters? ” And only then went into details. We understood that each question not only allows us to find out what the respondent understood, but also tells him which way to think.

Clarifying questions . You often have to ask the respondent to clarify or complement his thought. You can repeat a part of the phrase and ask the respondent to clarify what was said: “This was not clear. “What exactly is incomprehensible?” You can find several ways to ask clarifying questions in an article on the Nielsen Norman Group website . In particular, a curious Colombo method is described there: when the moderator specifically uses imperfect language and unfinished phrases to motivate the respondent to reveal the topic in more detail.

Sometimes respondents contradict themselves or simply do not speak very clearly. One of the favorite tricks of our moderators is the phrase “Do I understand correctly that ...?” So you formulate what you understand from the respondent’s words, and check if this is true.

Questions about the future . Very often we would like to ask the respondent: “Will you use this opportunity?”, “What will you do in such a situation?” , Etc. Unfortunately, you should not believe the answers to such questions. And not because people want to deceive you. They just don't appreciate their potential behavior. The test can show that all users are happy with the new functionality and are eager to apply it immediately, but the implementation statistics will not be so rosy. Therefore, be interested in past experience. If we are talking about a new functional, ask: “Were there situations in your life when this might be useful to you? What? ”,“ What problems would it solve? Remember the last time you encountered a similar problem . ” Ask for specific examples. The respondent may feel that the functionality is necessary, but it turns out that he personally has never needed this before.

Questions from the customer . When observing the test, the project team sometimes has additional questions that are not considered in the script. This is completely normal and even good. Sometimes designers or managers break to run to the respondent (at least after the test) and talk to him personally. However, it is better if all additional questions go through the moderator: customers may not have the skill to ask them correctly. We always offer the project team, who oversees testing, to broadcast questions to the moderator using the messenger. Very often, questions have to be reformulated on the fly. And in some cases, the moderator decides to ask them at another time, or even not to ask at all.

Tips

Although one of the moderator's tasks is not to prompt the respondents how to solve problems in the interface, sometimes it is necessary to prompt. I remember how I led a test on one of my first projects and, gritting my teeth, for a very long time did not say anything to an inexperienced respondent. As a result, the test instead of 45 minutes took two hours. But the saddest thing is that I did not receive any additional knowledge from the exhausted person.

Prompt or interrupt? Tell only if the continuation of the task after the help brings additional knowledge. For example, a task consists of three steps, and the respondent cannot complete the first step; you need to prompt, as you move on and check the remaining block of the interface. If the task is small and its result is not tied to other parts of the test, then instead of prompting it is better to invite the participant to proceed to the next task. If the respondent asks to explain to him how to act, do not explain it better. If the participant insists, I usually suggest doing it after the test. So you leave the respondent a chance to encounter the problem himself while further working with the system, and your explanation will not affect the image of the project that has formed in his head.

When to suggest? The respondent himself can ask for a hint after the first unsuccessful attempt. In this case, it is better to say: "Let's try to look again . " Before prompting, you should make sure that the person tried several options, looked in different places, but really did not find anything. This moment is individual for everyone. Someone is ready to try different ways for a long time, someone gives up at once. The main thing - do not allow excessive frustration of the respondent.

How to prompt? If you see that it is precisely time to intervene, then use the following strategy. Do not prompt immediately and clearly. At first, just direct the respondent in the right direction, for example: "Let's try to look for this function in another section" or "It seems to me that we are not allowed further because there is a problem with some field . " Only if the general help does not help the respondent, tell me specifically. But carefully, watch the words. Do not enter concepts that the test participant has not yet understood, do not use complicated vocabulary, and avoid generalizations about the work of the system that may affect the respondent’s opinion about the product.

Challenging respondents

You know how and when to ask the right questions, you can easily handle the technology, you have studied the project far and wide, but then you come across it - a difficult respondent. And everything is not glued. I tried to highlight the most vivid examples of complex behavior that the moderator will have to correct.

Silent

How to recognize? Answers to all questions almost monosyllabically. If you use the “thinking out loud” method, it always forgets about the need to comment and again falls into silence. In fact, the test is not the worst respondent. Much worse to get this in-depth interview.

What to do? Perhaps you simply did not initially establish contact with the respondent, try to communicate with him. Smile, change the tone; Maybe you are tired and ask questions too formally. Or is it just a feature of the nature of your interlocutor. In this case, be prepared that you will have to “unwind” the respondent all the time. Having received a brief answer to the question, ask specifying: “why?”, “Why?”, “In what situation did you encounter this in life?” , Etc. There will always have to think what else can flow from the answers so that pick a suitable question.

Chatterbox

How to recognize? At first, he may seem like the perfect respondent: contact, easily responding. But now half the time has passed, but you still can’t finish the introductory interview. One of our respondents began to answer the question of experience with the words: “When I was little ...”

What to do? Watch the time and do not let the respondent get stuck on topics irrelevant to the study. Here it is important to be polite, so as not to offend a person who sincerely wants to share with you, but firm, otherwise you just do not have time. I usually use the phrase: “I'm sorry to interrupt you. It seems to me, I generally understood what you mean. Let's move on, we still have a lot of questions. ”

Expert

How to recognize? Throughout the test, you hear tips from him: what to change, what colors to use, where to move the function ... All users are happy to offer you ideas on how to improve the product, but the “expert” has all the emphasis in testing shifting to evaluation and discussion. You get a lot of irrelevant ideas, because users are not professional designers, they are not aware of the project features. Even the respondents often do not plunge deeply into the tasks, as their eyes always shackle the next minor imperfections of the site: the color is not the right one, the field should be placed to the right, and so on.

What to do? Sometimes this behavior is a consequence of improper accents during the introductory briefing. When the participant decided that they called him to evaluate , and not to test the product. The edge is quite thin, but it is. Nevertheless, no matter how correctly you choose the words, you can get a respondent who perceives the meaning of testing precisely as an assessment of each element of the interface and a suggestion of solutions.

When the “expert” offers another idea, it is important to understand what is behind it: a real need or just a desire to speak out. For example, a mobile mail user says: “The function to forward a letter needs to be removed from here . ” If the moderator 's question “why?” He answers: “Because I always accidentally hit her with my finger when I respond to the letter,” this indicates a problem and a real need. And you will most likely solve this problem differently, and not just remove the function. If the “expert” is not able to clearly answer why he needs change, this is not a problem and not a need. Therefore, always ask why and why; or even better, ask the “expert” to recall an example from life when he really needed what he offered.

If the “expert” is too immersed in some idea or lack of an interface, which can unproductively increase the testing time, do the same with the “talker”. Gently interrupt and continue to perform tasks. In general, it is better to switch such people to activities on the test, although they will always be happy to return to the discussion.

Also, such behavior should be expected from truly experts in IT: developers, managers, designers. They cannot abstract from their experience and begin to issue an expert assessment of the project. This is good and useful, but not in the framework of usability testing, since it has nothing to do with user experience. Therefore, these people just do not need to be invited to the tests. You can read more about the principle of selection of respondents in my article on recruiting .

Zashuganny

How to recognize? You have just seen how the respondent suffered and suffered while performing tasks, but during the discussion he asserts: everything suits him in the system, no problems arose ...

When testing company employees (for example, for tests of internal IT systems, interfaces of call center operators, etc.) the following is possible. Respondents are afraid that their career depends on what is happening on the test. They perceive the test as a test of their skills and competencies, prepare in advance (and possibly consult with other people who have already participated in testing) and try to complete the tasks as well as possible. In addition, fearing the wrath of their superiors, respondents can hardly speak of product flaws and actively praise it, even if the whole test suffered.

What to do? Try to convince the respondents that the test is not aimed at testing their abilities, the authorities will not receive the results, and the project really needs comments, including negative ones. But, unfortunately, even if you speak it in the introductory interview, you may still not believe it. Working with this problem is not even at the moderator level, but at the level of the entire research department and their customers. Not bad, if management sends out to the whole company promo videos about the benefits of research and how employees can be useful for projects. It is also good to show the results of at least the first research on the company (these are the problems found, so we plan to correct them).

If you are only at the beginning of a way to increase the awareness of employees about internal research, then the “shaky” cannot be avoided. You will not be able to completely relieve stress - therefore do not trust the value judgments and satisfaction questionnaires too much. Look at the activity during the test more: even if the employee says that everything is fine, the problem can be seen. In addition, it is important for you to ensure that respondents do not prepare for testing. Therefore, do not tell in detail about the essence of testing in advance, and ask the test participants not to transmit information about the tasks to their colleagues.

Uncertain

How to recognize? As he completes the assignment, he is always seeking approval from the moderator. He often turns around and asks questions: “Am I right?”, “Can it be done now?”, “Should I continue?” , Etc.

What to do? If you encourage him once, he will continue to seek your support. In addition, such encouragement is a hint. As is the case with any questions from respondents, you can answer with the phrase: “What do you think?” . Sometimes I even pretend at first that I didn’t hear the question. Often the person after that just continues the task, and it turns out - it was a question in the air. If you see that the respondent is not at all sure of himself, you should stop and say something like this: “Everything you do is a priori correct. Here you can not break something or spoil it. - , , , ».

? , ; , . , , . , , . , .

What to do? , , , . , «» . — , , , . , . , .

? , . , . - .

What to do? — . . , . . , : , , . - , , : ( ) , ( ) .

? . «», , , - . , - . , , , .

What to do? , . - . - ( ). , , . : « , . . , , . - , ». , . : , , , , .

, . . . , . , . , . — .

Conclusion

You become a good moderator only with practice. But even experienced moderators are wrong. Therefore, it is important not to weaken self-control. Be as neutral as possible, do not ask too many questions, carefully select words, watch your non-verbal signals. Sometimes I specifically review video tests and note my mistakes. It is also useful to work in a pair with a research colleague, paying attention to the wording of questions, the reaction of respondents, prompting each other. This allows you to stay in good shape and not lose the skill. Good tests to you!