User interview by internal company forces: through mistakes to discoveries

Hi, I am Sasha, a scrum master from Tutu.ru in the team of tours. Not so long ago, we conducted a user interview for our new feature - Jarvel. I want to share with you the mistakes and discoveries that helped us conduct this interview in such a way that it touched and changed the cornerstone of the feature. This helped the product and pumped our team, I hope it will be useful and interesting for you. But in order to understand the essence of our discoveries, you need to be introduced to Jarvel and lead us along the way.

Hello, world!

Some time ago we had an idea - to create a virtual assistant who will help users to choose the tour that suits their requests, which:

• like a person, it will give an opportunity in informal communication to indicate your Wishlist;
• helps to identify and express those Wishlist that the client has not even formulated for himself;
• as a real assistant, he will search not for one, but for all countries at once, reducing the huge issue and narrowing down the choice to several ideally suitable offers;
• and will not just gradually reduce the output, as usual filters, but give the client control and understanding. For example, parents Julia and Peter go to the sea with a five-year-old Anton. In this case, the assistant will offer to remove extremely hot countries that will be uncomfortable for the child, and leave only hotels with a sandy beach, excluding pebble. In this case, the assistant will not only cut off these proposals, but will explain his position and will enable Julia and Pete to cancel such filtering.

')

Hmm ... assistant ... and then we remembered Tony Stark's artificial intelligence - Jarvis. And since we are engaged in tourism and travel, i.e. the theme of travel, the name was born by itself - Jarvel (Jarvis + travel).





www.filmpro.ru/materials/26976

TOP SECRET. Insider information

I will reveal one of the main secrets of domestic cuisine. In order to form the issue of the actual proposals, we need to send requests to the tour operators, and this is a very slow case. Initially, when we only developed the idea of ​​an assistant, we understood that searching for several countries at once (namely, Jarvel is doing this, choosing the right tour for you not only for one country, but for all of you who are interested) can take quite a long time. Then we came up with a solution that seemed to work well for the prototype: let's not carry out real searches, but search for our own cache (that is, we will search for the results of recent searches of other users on the site). Such a search goes many times faster, and the client will not wait. And to maintain relevance, we will use the cache is not older than an hour. In the future, we planned to work on the cache system, fill it with more complex algorithms, increase the storage time, introduce relevance coefficients, etc. There is only one problem with this. If no one, for example, in the last hour has looked for tours to the Dominican Republic for the dates you are interested in, then you will not see the Dominican Republic in Jarvel’s extradition. Knowing this, we still were guided by what we then considered the main convenience of the client - the quick issuance of Jarvel's instant answers. And we would also provide a full choice. Therefore - cache.



NB This solution will still fire.

Not everyone sees Jarvel the same way. The reasons for the interview

The assistant, starting from the prototype, was developed iteratively: we collected and analyzed metrics, conducted corridor tests among other development teams, organized demonstrations with feedback gathering for colleagues and customers. And then one day we invited a non-IT professional Kolya, an employee of our reception, to a similar interview. . And ... it was a fiasco, bro. Kohl simply did not notice any of our innovations, and with many of the old ones that we introduced earlier, Kohl interacted in absolutely the opposite way - not at all in the way we had planned.

Then we thought.





www.meme-arsenal.com/create/template/747344



And what if Kohl is not a minority, but the majority of our future users?



No, I repeat, we collected the metrics, analyzed, conducted research and understood that there are some problems.



True, there was one “but” that we did not take into account, and thanks to Kolya we realized that all these people we interviewed were IT specialists, like us, and their gaze could eventually become very blinkered ...



Therefore, we made a decision that it was time to turn to users of another type: non-IT students and those who buy tours offline. After all, one of our goals is to show that buying tours online is just as convenient as offline.

Preparation: people

We decided not to hire an outside specialist for in-depth interviews with our user base (colleagues from neighboring teams had previously worked with him).

We ourselves wanted to penetrate the spirit of Jarvel, so that developers, testers, analysts and designers could see people-users alive, hear and feel their pains, start thinking a little more as users, not as creators.



Not only IT-specialists work in Tutu.ru, we, of course, have many departments: HR, accounting, lawyers, HR management, marketing, etc. Previously, the team already had experience in the use of internal resources: we interviewed tutushnikov-parents when developing themes for holidays with children. Parents burned and participated very actively.



And although we had previously conducted some similarities of interviews, but still, in fact, we were not the most experienced interviewers. But if we call "our own tushushnik", and not users from the outside world, then the fear of this inexperience decreases.





v1rt.ru/memy/nelzya-prosto-tak-vzyat-i-voyti



Therefore, for a user interview, we used internal resources:

a) parents who have already proven themselves in previous surveys (of which we singled out “non-IT students”);

b) department employees who do not belong to the developers: they simply went around the office and offered to undergo an interview.



So we followed the company's credo: we tried to achieve more with less.

Preparation: questionnaire

In parallel with the formation of the list of interviewees, we began to prepare a questionnaire and work out the format of the interview. The basis of this process was laid by the book of Robert Fitzpatrick “Ask Mom: How to communicate with customers and confirm the correctness of their business ideas, if everyone is lying around?”





www.ozon.ru/context/detail/id/140446253



In short: talk less about yourself (don’t chat at all), let the client talk about his life in context, go away from the problem, determine the motive, focus on the client’s past, and not on his statements about what will happen in the future. If the client says that he or she needs one or another function, specify how he is doing without it now, what is she doing to solve this pain. From the answer you will understand not only the way of thinking and actions of the client, but also whether such a feature is really key and painful, and not just a Wishlist, without which it lived, lives, and in general it is not particularly needed.



Prepare for a meeting in advance and plan ahead what kind of three important answers you want to receive from any particular category of people. Be sure to ask a question that is scary to hear the answer. Without such a question, you simply amuse your self-esteem, and do not conduct real research.



Never be content with the answer that all is well. Search and strive for the mundane: for further steps, questions, contacts. Empty compliments are an ideal foundation for self-deception castles in the air, which can lead to the creation of a product that no one needs.



So, it was necessary to determine the key issues. For us it is:



1. Is it easy for clients to interact with Jarvel?

2. Does he get what he came for?

3. What subjects are important to people?



And the same question that needs to be asked, but to get an answer to it is scary:



4. Is Jarvel needed at all? Right?



Asking these questions directly would not lead us to the true answers of users. We felt that if we did not interfere and simply let the interviewees interact with the product and ask for comments on what is happening, then we will easily find answers to the first two questions (is it easy to interact, does it receive what came from). Accordingly, everything is on the ointment.







With the third question, about the necessary subjects, it was more difficult. A simple answer from the user (for example, “It would be great if you added a question about stardom”) is actually not enough. Like a book, it is important to validate the value of such information through past customer experience. Therefore, we agreed that during the interview we will collect all such Wishlist, and then we will ask the guys to tell us about their latest experience of buying tours. This is how we can find out the real requests and desires of the client. During the story, we will have to “return” the Wishlist to the client and ask: “Vasya, tell me, but when you went to Turkey, how did you determine the star of the hotel you went to?” If the Wishlist really came from Vasya's needs, He will tell you how to select hotels with a star. Otherwise, Vasya will answer: “Yes, I somehow didn’t look at it, it was more important that the hotel was near the sea”. "Aha" - we will answer, we will note this Wishlist, but we will leave it with a lower priority.



As for the last “terrible” question, here we decided that if after the “poking” stage, we ask the client about the latest experience, return him to his Wishlist, this will allow him to abstract himself a little from Jarvel and concentrate on himself. And then we will ask him: “Vasya, how will you plan your future vacation?”. If, in response, we do not hear anything about Jarvel, it means that now he hasn’t gotten into the client’s need.

Preparation: technical equipment

1. Devices

Jarvel currently lives only on the mobile version of the site, so we gave our respondents a phone and displayed the image on the screen so that the guys could interact with Jarvel directly, and we could see all their actions



2. Video Recording

We organized a video of all the interviews, because:

a) Considering the product owner, scrum master, designer and analysts, we have 8 people in the team. To make the interviewee feel comfortable, it is recommended that the interviewers should have no more than 2-3 people. Therefore, we had a rotation of interviewers. The video was needed if something interesting had to be seen by everyone.

b) In addition, the video is indispensable in deciphering the interview, in clarifying incomprehensible details.





www.vibe.com/photos/roll-safe-memes



3. Environment



All three questions from the methodology, which we talked about earlier, essentially focused on whether the UX (user experience) assistant is convenient for users. And remembering the cache, in order to ensure the most complete interaction with all the features of Jarvel, we made the decision: let us run every hour on the site before the interview itself multiple searches in different countries and on different dates using our automated tests. Thus, we fill the cache and users will have a large selection ... We made a mistake, and that changed everything.







When I transferred information from this draft to a copybook, I forgot the point about searches. They were not in the final document either, and during the interview itself, too.

Interview results and great discovery

Using all the above, we conducted an interview. I can say that you are pumping heavily in conducting an interview, in tenacity and attentiveness even after one or two times. But collecting information is only half the battle, now it was to analyze it. We created a document where we merged all comments / information from all users and divided this information into steps. Here, of course, videos were very useful for clarification. Then we identified pain points and collected the frequency of their appearance among the interviewees. And it was here that forgetfulness fired, thanks to which we made the main conclusion: “we have very little issue or lack of it”. After all, I forgot to transfer information about the need to run autotests from a draft to our final version, and as a result, they were never launched. And this in turn allowed our clients to face our real issue, and the team to understand how bad everything is in reality. 80% of our respondents in one form or another had problems with issuing, the reason was the approach that we initially chose to search for - searching for sentences stored in the cache. As a result of the interview, we found out: it is convenient for users to interact with Jarvel, they are pleased and they are benefited by questions that allow you to control the issue and identify their desires, visualization and working with several countries at once allows you to see and think about countries that you never thought about. However, all this does not make sense if the issue is empty or consists of several uninteresting sentences. But what to do with this is not clear.





www.drive2.ru/l/524851876592616104

Admission fact

Not everything you encounter can be changed,

but do not change anything until you face it.

James baldwin



From recognizing that which is undoubtedly present, comes a certain freedom.

After that recognition comes knowledge of how to act.

Gregory Bateson



The interviews took place on Monday and Tuesday, and by Wednesday we had an analyst with this indisputable fact. We held a meeting at which we decided that we should concentrate on the issue of issuance and by Friday gather technical analytics, work out possible solutions, and put all the dots on i.

As a result, we completely revised the entire search algorithm and the order of questions of Jarvel. We realized that we could not get away from a long search anywhere, but:

a) we can speed up the search, eliminating from it irrelevant to the client country. That is, to build a client's communication with Jarvel so that we can eliminate those countries that do not fall under the wishes of the client before the search starts;

b) to turn the waiting process itself into something that benefits the client. Then we remembered Instagram, Wikipedia, and downloads screens for computer games, such as Neverwinter Nights with its management tips, or Bayonetta with a free zone of combining strikes. And we decided to combine these ideas into one, creating a tape with interesting facts that can be liked and in the future prepare the person for subsequent questions. For example, before choosing a star, tell me that two stars in Turkey are not equal to two stars in Thailand.

Eventually

This is a story about an interview that we, as a team, decided to conduct on our own, and this interview transformed the ultimate basis of our feature, although initially we were preparing to collect problems that concerned only UX. We were mistaken in trying to achieve maximum authenticity in one aspect, and not paying attention to the fact that we are deceiving the user-generated results and, ultimately, ourselves, artificially recreated through auto-tests.

Conclusion: user interviews, even in a small format, even among their colleagues, can be of immense benefit, pumping the team in understanding people and highlighting the blind spots, if carried out in conditions as close as possible to the real ones, and not only in the aspect that you research, and in general. Choose questions to which you want to get an answer, but do not let them violate the “reality” of the user experience. Be as authentic as possible, communicate with users on your own and the results may surprise you.

PS Tips and finds from the interview

• It is known that many sound typing can be annoying, so we agreed that we keep records with a pen on paper.
• Our small team of two or three people shared the following: one - the leading interviewer, his goal is to focus as much as possible on a person’s behavior, on his facial expressions, on how he speaks, on his gestures; the second is the secretary, his goal is to record the words and actions of the client. Both can make notes, both can ask questions, but everyone has their own focus. The main goal is not to interfere or prompt the client, to see the situation as close as possible to reality.

memesmix.net/meme/rbh3d3

• During the interview it is necessary to avoid ambiguity. For example, one of the interviewees said: "Now he talks a lot." Such comments should not be left unclear. In this case, we were asked to reveal what exactly she had in mind: “Well, for example, I want to change the number of nights, I just can poke and change, and not this“ now we will change the number of nights. ” for change, I already understand what I am doing. Why do I need additional information about what I am doing? Time is wasted, because everything is already clear. ”We learned not just an emotion, but got to the bottom of the reason that worries the client.
• We read about it and realized from our own experience that it is extremely important to conduct a short recap after each interview. We gathered, discussed what we heard: everyone shared what hooked him; they determined the growth points for Jarvel, which we knew about, and those that we didn’t even guess. For example, they brought in such ideas: to mark not only traditional weekends, but also holidays in the calendar, or to add a new type of rest - “lie on the belly and do nothing”. It was discussed that somewhere they didn’t keep a pause, but somewhere too early they began to “return” to the client their Wishlist, or that it’s extremely important to follow the client’s fingers during the interview - how he clicks on the screen, because the monitor that broadcasts Us user action does not show "empty" clicks when the user does not hit the screen elements the first time.
• With that, we tested the interface of Jarvel, for us it was very important not to stop the client, even if he had already switched to the purchase, because Jarvel is just a convenient way to enter your wishes or to identify them. The ultimate goal is to give a person the impression, sell him the best suitable tour for him. And if the client falls off at some of the steps after Jarvel, it is extremely important for us to know where and how.

Thank you very much for reading. If you have questions - ask them in the comments, I’ll be happy to answer. And look at Jarvel 1.0 (for now only on mobile phones).