Focus groups for user research: participant's impressions, criticism and adaptation of the method

Shot from the series "Silicon Valley", a scene with a damaged focus group.

When it comes to Customer development, or marketing and UX research, focus groups are often mentioned. More often they are criticized, or ridiculed. And usually fair. I almost did not meet situations when the focus group method was correctly chosen for the tasks. In terms of the frequency of wrong choices, focus groups only catch up with problem interviews and polls. Therefore, it would be more correct to criticize not the method, but its inconsistency with the objectives of the study.

Recently, I became a participant in the study of new on-line services. The event was announced as a focus group, but all requirements for focus groups were violated. This fact could have been the reason for the article, but it turned out that just due to these violations, the method turned out to be quite sensitive for testing the hypotheses about the users' perception of the products.

In the article I will tell

- how the study was organized,
- what I felt as a participant,
- what should be improved in this method
- for what research method will be useful.

The article will be interesting:

- product creators
- product managers,
- to marketers,
- custdev and ux-explorers.

How it all began

On the page of my friend Andrei Karpenko on Facebook appeared
The guests were not paid for the participation, but there were many who wished, and the organizers additionally rented a large hall.

How did it go

We gathered in the hall with a screen and rows of chairs. About 50 people came, many were familiar with each other. In a welcoming speech, the moderator said that we will evaluate 5 services for travelers and asked which of them we are already familiar with.

A list of services appeared on the screen:
1. tripadvisor.ru
2. vivaster.com
3. malpatravel.com
4. [new service]
5. onetwotrip.com

This was how the study went: rows of chairs, a moderator and a large screen.

Participants began shouting out services they knew. After the moderator voiced the rules: we will be shown each service in turn and will distribute questionnaires in which it will be necessary to evaluate the impressions of the service on a 5-point scale, and then express our opinion. We were asked not only to put a mark, but to clarify the opinion out loud, or in the questionnaire.


Such forms were distributed after the demonstration of each service.

Then everything went pretty quickly: the site page was loaded on the screen, the assistant moderator scrolled it down and clicked on any link. At the request of the participants sometimes returned, or entered a query in the search bar. A few more people handed out questionnaires and collected completed. The most active participants loudly expressed their opinions, the moderator asked questions and asked the opinions of others. What is happening was shooting an open video camera.

Sometimes, in order to induce silent participants to express an opinion, the moderator would ask: "Who put the" 1 ", who put the" 2 ", who put the" 3 "... tell us why." It worked well, and people expressed themselves more willingly.

The atmosphere was good, people got involved, and almost everyone spoke closer to the end. After the graduation, the participants did not disperse for a long time, were excited, and came to meet each other. Entrepreneurs and owners of IT products told the moderator that it was very useful for them to participate in it, the guests asked if something similar could be done to evaluate their products.

What was done well

1. The combination of quantitative and qualitative methods is quite convenient and was easily perceived by the participants.
2. The questionnaire was simple and short, with explanations to the questions.
3. The moderator performed his task well. He did not betray the personal interest and interest of his customers, did not allow to deviate from the topic, supported each statement and delicately involved the silent participants to speak.
4. The organizers were friendly and created a safe, friendly atmosphere.
5. The event was free. This does not kill the work environment and is good for enticing participants. Product owners should not be afraid that no one will agree to such research for free. This is not true.
6. A lot of people came to meet the moderator, they were active and ready to help. So, time was not spent on warming up and recruiting participants, but it influenced the results, I will write about it below.

My difficulties as a participant

1. Assistant presenters quickly loaded and flipped through sites. I saw many services for the first time, and for me this browsing speed was not typical. When participants asked to unscrew it, this was done quickly again. It was only when the discussion began that it was possible to carefully read some headlines. It was not possible to click through the links, to see the inscriptions. But the organizers did this if they were asked. Because of this speed, I had to rate on the first impression, often only later I learned that I did not fully understand the concept of the site.

2. Questionnaires for assessments were distributed by turns and immediately taken away after evaluation. When other services appeared, I wanted to change the estimates of the previous ones, but I could no longer return my profile. Each new service was evaluated under the impression of the previous ones. The first ones did not seem so original and convenient to me, but I could not change the grades, but I evaluated the next ones more strictly.

3. I was not clear how to answer some of the questionnaire questions. For example, the question about trust: “How does a service form a trust relationship: trust of information, payment system, quality of services”. In my view, trust is a dichotomy, not a continuous variable. That is, I either trust the service my money, or not. Trusting “to fours” is not quite clear how, and what my consumer behavior will cause. But these are my impressions, the questionnaire did not prevent me from putting "1", or "5".

4. Active participants almost immediately began to speak. And they did it even before I understood what the service was about and put a mark on it. They spoke most often critically, referring to their experience, talking about other services. These active participants had much more experience in using services for travelers than me. Under their confident statements, I changed my opinion several times and instead of “5” I put “3” or “2”.

5. Some functions of the services, or errors in the texts on the sites, I did not notice until they were called by other participants. After their words, my opinion changed, and I could no longer perceive the service as I did before. I'm not sure that I myself would pay attention to the same functions, or errors, and would refuse the service. Now I do not know.

6. On one site, I noticed that the title promises to search for events and asked to see whether the information is really grouped by event. Then I asked to enter an event in search, for example, “carnival”, or “city celebration”. But it turned out that the site had only tours through the cities, and the title was misleading. At this moment, it seems to me, I influenced the ratings of the other participants, because earlier they did not notice this discrepancy and praised the site for the unique presentation.

7. I had the opportunity to speak, and several times I raised my hand. But not always my statements were heard by the other participants, the dialogue did not work. And so I often remained silent, even when I did not agree. Due to the fact that there were a lot of people, it seemed that the audience agreed with the 2-3 loudest critics.

8. As the creator of an IT product, I was generally loyal to what was shown to us, because I understood what kind of work was behind all this. Sometimes it seemed to me that the most active critics have no such experience and find fault with the font and the background without trying to understand what the goals are solved by the design. And from this heterogeneity of experience and preparation of the audience, too, often chose to remain silent than try to enter into discussions.

9. At some point, when I said loud participants, I realized that they misunderstood the question of uniqueness: “How unique is this service in its concept and offer to the client”. They criticized the secondary design of sites, but did not talk about the uniqueness of the proposal for the client, which contained the site. Then I began to unscrew the rest of their critical statements that influenced my assessments, and realized that perhaps they had not previously understood the essence of the services, but made statements only about the appearance of the landing pages (and then it seemed to me that they were more than understand these services). All this finally demotivated me, and I no longer listened to anyone and did not try to argue with them.

What can be improved to get more relevant information

Since this event didn’t meet almost any requirement for focus groups, I’ll distract from the comparison and will discuss it as a separate method of collecting information from users. But the basic requirements for focus groups will be true here too, so I’ll also mention them.

Members

1. It is better if the participants of the event are not familiar with each other and the moderator. It protects against the transfer of roles, less impact on the statements, social desirability, support for others and the fear of appearing inexperienced and stupid.

2. To understand how the service is perceived by people who are expected to purchase, it is necessary to invite such people to the research. If we are talking about new products, then people from the group of potential “early followers” ​​should participate. This means that prior to the study, it is necessary not only to clearly define the target audience, but also to narrow it down to early followers (that is, people who are much earlier than the early majority will be ready to look at the product and start using it). The filters used in the invitation post created an overly wide audience with different experiences using on-line travel services.

3. Members must still be less. To hear everyone and get a discussion, you must either reduce the number of people (up to about 10 ±) and the number of questions, or significantly increase the time.

4. In order to achieve deeper statements and understand how people perceive the product, you need to select participants with similar experience and training. So they will have more desire to convey their thoughts and there will be no feeling that there is "no one to talk to" and "no one to prove." (In the practice of group interviews and focus groups, they sometimes deliberately achieve age and gender homogeneity, and sometimes, conversely, reduce participants with different experiences in order to get more information in the discussion - this depends on the tasks and experience of the moderator).

Research mechanics

5. If you need to understand how people perceive the service in natural conditions, it is worth giving them the opportunity to look at the sites themselves with the usual speed: on phones, tablets, or computers. It is easy to organize and can be combined with studies of navigation.

6. If you need to know about the reaction of users to the product, you need to give them the opportunity to make a decision and give an assessment before the active participants begin to speak. To do this, enter the minutes of complete silence to fill out the questionnaire, and then start the discussion.

7. It is worth provoking an active discussion on the topic between the participants. This is necessary in order to better consider what thoughts arise in the minds of users, and how they make decisions. In single statements on the answer “Why?”, Short and slender rationalizations are usually given. But in the discussion the idea turns around, the participants feel more secure, are distracted by the task of defending their opinions and reduce control over the harmony and social desirability of statements. At this point you can get useful content.

8. The discussion and controlled disputes of the participants also help to provoke a more thorough immersion in the service and find out which other points users decide. Such information usually does not appear in the first responses.

9. Discussion and joint decision making should also be provoked when you need to evaluate how people organize teamwork when using a product, or when making a purchase decision.

10. In order to reduce the influence of the order of the services being discussed on how they are perceived, it is worth keeping the participants' questionnaires until they can see all the services and fully compare them with each other. Since comparisons are still not avoided and spontaneous reactions can only be on the first service, it is worthwhile to balance them in rights. Or you can hold several different events for different participants, at which different services will be demonstrated first.

Processing of questionnaires

11. It is possible to change the procedure for collecting questionnaires so as to divide the marks made on the first impression, before the influence of the statements of high-profile participants, and after such influence. The way the questionnaires were collected now does not make it clear whether these estimates were spontaneous and how much someone else’s opinion influenced them.

12. You can also add the question: “Has your opinion changed after the discussion?” And “Which arguments influenced your opinion” - this will allow you to investigate which arguments of the commentators turn out to be the most significant when deciding and which of them are able to overcome the user's first impression.

13. It is necessary to assign participants some ID, if you want to preserve their anonymity. So the questionnaires will not fall into some unparsed pile of average values ​​and it will be possible to track trends in the estimates of people with different experiences.

14. It is necessary to include the minimum passport: questions of the social-dem block (gender, age, profession, experience, etc.) - so when processing the questionnaires, it will be possible to identify segments of users for which some product functions are more important than others. This means that developers will receive not averaged values, but differences in the perception of services by users from different segments. But all this makes sense if the questionnaires will be collected quite a lot, and the study will acquire a quantitative component.

Interpretation of results

15. It is worth clearly separating users' idle sayings about design (colors, fonts, illustrations) from their readiness for action. Praise and high marks on all items of the questionnaire do not guarantee that the participant will be ready to use the service, or recommend it to friends. And vice versa. This is important to explain to the owners of the products that will see the discussion. They are often ready to immediately perceive criticism as a team to change something, although this is not always related to the behavior of their target users and in general is not always required to be done.

For what tasks is this method suitable

1. In the variant with fast scrolling of sites and loud statements of active participants in the group, the method gives a good impression of the user's first reaction. Usually, users are not loyal enough to give the product a second chance and carefully understand its unique offer. If they do not see the value of the service in the first fast scrolling, they can quickly leave and not return. With the help of this moment, you can understand what thoughts fly through the user's head right before he closes the site page.

2. Loud active commentators of the service add “weighting”, as they help determine whether the user manages to quickly grasp the essence of the service, or get carried away by its design, despite the strong interference and influence of the opinions of others.

3. By supporting the statements of the first active critics, one can also track which criticism from the side of the users is more likely to trust, and which is considered just personal experience that has nothing to do with it.

Conclusion

1. The event I visited turned out to be an improvised group discussion of on-line services using questionnaires and usability testing elements.

2. If the method is not changed, it can give an idea of ​​how users perceive the service at a glance, and how they grasp its value proposition.

3. If you do not change the procedure for collecting questionnaires and do not add social-dem. questions, we get very averaged data with a large scatter of values, reflecting partially spontaneous, partially changed opinions. Conclusions based on such data is not worth it.

4. If you refine the selection of participants and the procedure for discussion and collection of questionnaires, you can get an informative version for the study of the product, a combination of qualitative and quantitative methods.

5. For owners of the product can be useful any options for this method. It is only important to separate the statements of the participants of the event from their real consumer behavior and clearly determine whether they belong to the group of early product followers.

(The article was published with the consent of the organizers of the study).