MailChimp UX Team: How We Do Research [Part 3 of the book]

[ Tl; dr ]

[ 1st part of the book ]
[ 2nd part of the book ]
[ 4th part of the book ]
[ 5th part of the book ]
[ 6th part of the book ]
[ 7th part of the book ]
[ 8th part of the book ]
')

Data radicalization

Gregg Bernstein

One of the basic principles in working with content or design of something is to moderate the emphasis. By overloading a design or textual material with a mass of prominent details, you risk completely knocking your audience’s attention away from significant moments for you. Working as a researcher at MailChimp, I realized that our guys learned how to achieve good results using the completely opposite approach.

We begin with the fact that everything is important - each of the indicators has its own history. Following the path of such stories, combining them into a single whole, we come to an understanding of the directions of our research, highlight important points in them and set priorities.

Your customers are smarter than you

We do not put ourselves above those data and stories that we are trying to find in the course of research. We have many sources for obtaining initial data: customer feedback in the mail, the results of surveys and interviews, usability testing, analytics, etc. The absence of unnecessary haste in the formation of assumptions about the possible results of our research helps us to feel freer in working with data.

By unleashing the hands of our “inner” Colombo, without exaggerating our own importance and awareness, we consider each opinion as an expert, bearing in itself weight and meaning. This approach helps us to find dependencies in the studied data.

Every day we receive letters with complaints and wishes about the work of our services and read each of them. Why? We are sure that our customers know much more about the use of MailChimp services, and if we work for them, then why not learn from them?

Last year, we received only one request for the function of notes in the subscriber profile, but did not abandon thinking on this topic - we addressed this person directly, sorted out his use-case and made appropriate changes to the development plan. Having shown due attention even to a single customer request, we were able to help the many users of our services.

Similarly, we approach the analysis of interview results. For example, if the data gives us direction for reflection in the direction of eCommerce, we turn to customers and try to understand their typical and atypical workflow and their products. Understanding the context of our research helps us greatly in achieving meaningful results.

The ReWired Group offers a similar approach when customers “hire” products and services to solve various problems, and researchers try to understand the motives in making decisions about choosing a particular product or service. We always give expert status to reviews and suggestions of our clients, try to understand each specific situation and preach an open approach to working with data.

Test customer hypotheses

Of course, each indicator tells us about a certain dependence, which is not necessarily applicable. As soon as we have a sufficient amount of disproving information, we reject the false hypothesis and move on.

For example, once we found data indicating that there is an opportunity to unsubscribe from a mailing list by accidentally clicking on the appropriate link. Having dealt with the source of the data, the frequency of occurrence of such a situation and the context, we decided not to continue the detailed consideration of this case. Possible options for finalizing the application could solve the problem of a couple of people, but at the same time complicate the work of a thousand other customers.

A detailed review of the indicators allows us to constantly learn from our customers and not miss the opportunity to improve our products.

Share your research

Larissa Wolfram-Gvas

Productive research requires multi-day preparation of tests, questionnaires and interviews, a pool of participants and pilot tests. The various stages of research take from several days to weeks, but in order to review the videotapes of the interview, prepare transcripts, screen out answers, conduct additional interviews and draw conclusions, even more is required.

What happens next? How do we translate research results into development plans? How to use research insights in practice?

Regardless of their interest, relevance and logical correctness, research should have a positive impact on our products and services. One of the main tasks of the staff responsible for conducting research is to report their [research] results to colleagues who make decisions about finalizing the design, developing a particular functionality, changing the approach to marketing and working with content, regulating technical support processes, etc. . All this constitutes the so-called user experience.

The study report is one of the generally accepted ways of disseminating results, which requires considerable time to study because of its size. We experimented a lot, trying to solve this problem, and came up with the following reporting options.

Posters

In 2013, we developed character posters that illustrate the main categories of our clients. Our UX-team worked on the characters themselves for several months, prepared a report with their detailed description and asked the designers to make the appropriate posters. We arranged these posters in the common room of the office in order to ultimately draw additional attention of colleagues to the people for whom we work. In the course of a large-scale redesign that we conducted a year ago, we also attached customer feedback to the respective posters.

Posters themselves cannot serve as a tool for a detailed study of the profile of our clients, but their main advantage lies in the speed of perception.

Office hours

On Friday mornings, we all gather together for a “ coffee with an expert ” - in the framework of such events, the speakers share their experience in organizing creative work, journalism, design finds and just anecdotes. Sometimes they are joined by our team of researchers, which shares the latest results of their work, taking the baton from previous speakers. Usually we are talking about general trends, covering the bulk of our customers, and therefore relevant to the entire company.

We share the results of such gatherings with the help of Google Drive or Evernote , in which we usually conduct most of the work related to research, but prefer to talk about the results in the course of personal communication.

In the course of joint lunches, which are less formal than “ coffee with experts, ” groups of 20 people and less gather together with designers and developers to view usability test videos. As part of these meetings, we are able to discuss issues related to design solutions in a relaxed atmosphere.

These ways of presenting the results of research are more dynamic than reports and make it possible to convey information to those whose attention cannot be concentrated in any other way except during direct conversation with the opportunity to ask questions and propose solutions.

Corporate Newsletter

Sometimes our researchers themselves use MailChimp to inform colleagues about the results of their work. This may be a topical collection of statistical information on the topic of indicators of the use of our services, integrated services or on the topic of user behavior. Usually such a compilation is supplied with brief comments or analysis.

Of course, this is not always enough for a formal presentation of data, but enough for a basic understanding. In addition, we attach links to additional sources on the topic of mailing.

Mini movies

Sometimes we mount short 5-minute videos that give our entire team an idea of ​​one day in the life of our customers. The customer faces, the understanding of their work space and technology helps to increase the sense of empathy. Customers' stories about how they use our product, what prevents them or, on the contrary, it seems useful, is perceived much better than standard reports.

Google Drive Reports

Of course, the distribution of reports is not our only way to "communicate", but one cannot do without it. We use Google Drive, with which you can simply share information and comments in asynchronous mode. The reports are read only by a part of the company's employees, first of all the management gets acquainted with them, which recommends the materials to specific performers in accordance with their projects.

Our designers and developers note that the reports work well as a reference in the process of working on a particular project. Usually our reports are rich in details and require careful reading - they receive this mark if necessary.

Evangelists research

Each of the reporting forms for our research solves its problems, and the MailChimp team of UX researchers is constantly working to disseminate the results of their work in accordance with the preferred channels of information consumption by company employees. One of the elements of our work is the informal discussion of research - the introduction of point remarks in the discussion of working moments with colleagues. Using this technique, you should pay additional attention to the accuracy, timeliness and brevity of your comments - you should not monopolize the entire discussion, reporting on all aspects of your research.

We try to maintain an open-air atmosphere at MailChimp by welcoming the free discussion of ideas, and the interdisciplinary nature of our research allows us to understand the impact of one team’s work on other employees. Discussing our work with colleagues, we share not only knowledge, but also try to bring people together who can solve problems more effectively by working together. For example, we know that our brand manager received interesting insights from our eCommerce clients, so it makes sense to discuss with him the relevant functionality that is at the finalization stage.

Research spirit

We are currently working on how to refresh the results of our research and attract more interest from colleagues. Especially for this, we attracted our designers and developed closed pages to present our results in a more attractive and concise form. In addition, together with the marketing team, we are working on a version with the presentation of all this also in the form of comics for our colleagues.

Our goal is to discuss the company's work with the whole team. We are trying to interest employees of the company in facts about the people for whom they work. Ultimately, we are committed to improving our products. Our team employs smart and talented people, and our task is to provide these people with relevant information that will help them in their decisions.

How we managed to process 506 thousand responses of respondents

Fernando Godina, Larissa Wolfram-Gvas

Feedback is the most important element in the work of the company, ensuring the adoption of informed decisions in the process of improving our products. In 2013, we decided to conduct an experiment and sent out a questionnaire with 46 questions. During the week we received a response from more than 11 thousand respondents. These are as many as 506 thousand responses that need to be processed.

Fernando: Primary treatment

We were able to quickly analyze the answers to multivariate questions thanks to the convenience of the specialized service SurveyMonkey, which allowed us to determine the share of mobile device use by companies with a certain number of employees or to assess the degree of use of the team approach to work in an industry.

We drew John Forman , who heads the research team, to study the data. John showed us a couple of his Excel techniques, but that's another story.

So, we were faced with the task of processing thousands of respondents' answers in a free form, and we decided to look at this task from the point of view of the work on assembling a big puzzle - we needed to compare data and isolate dependencies. Thus, we decided to classify answers with common features and instead of receiving 1966 answers to a question, we received 14 so-called baskets reflecting certain types of answers. Further, we ranked these groups, having received more specific answers to the questions posed, which could already be shared with colleagues.

Larissa: Search for meaning in chaos

Classifying answers helped us greatly in working with data, but in working with tens of thousands of answers, everything is much more complicated, and it is rather difficult to distinguish between them semantic dependencies. There are many approaches to working with answers in free form, but I will try to highlight the main points that will help you solve this problem using SurveyMonkey.

Decide what you would like to know.

When analyzing answers, focus on the main goal of your research. If you are trying to understand the number of your customers who are dissatisfied with any part of the functionality, your approach will be different from the one that you could use to determine the most frequently used functionality. Concentration on the purpose of the study will help to quickly process the data.

Classify data

As noted earlier, “open” questions are an excellent tool for gathering customer opinions, but with an increase in the number of answers to such questions, the analysis process becomes more complicated.

Of course, I always want to quickly find out the answers of our customers, and I immediately begin to scroll through the answers to satisfy my curiosity. Having finished with this, I get rid of those answers that do not carry any value in themselves - these can be either dashes or answers of those people who wanted to move quickly to the next question. Such answers fall into the "No Answer" box.

Next - I look for answers for common keywords depending on the purpose of my research. This may be MailChimp functionality (templates, user profiles, answering machine), user task (data import, segmentation, synchronization) or other software (Excel, EventBrite, various CRM). SurveyMonkey just allows you to search by keyword. In this way, you can quickly categorize your answers.

The SurveyMonkey test analyzer allows you to automatically find the words or phrases that are often used in the responses of respondents. This functionality does not give perfect results, but sometimes it helps to learn something new about your data and quickly enough to classify the answers. Then it remains only to process the answers that are not in the sample of the analyzer.

Refine your classification

After completing the classification, I look at the answers in each of the "baskets". Sometimes you have to refine, split or merge groups of answers. For example, the answer may be in the “interface” group, but in essence belong to the “content” group.

Decide on interview candidates

Surveying helps to find out what difficulties this or that group of users faces, but the reasons are not always so accessible for analysis. To complete the picture, our team of researchers usually conducts additional interviews with clients. Looking through the answers, I mark those respondents with whom it would be useful to conduct an interview, and highlight the questions with which it is worth sorting out the details. This should be done in the process of clarifying the classification.

Note exceptional cases.

Research helps to deal with the general trends that meet the majority of customers, but do not forget about those who stand out from the crowd. With such cases it is best to work individually, conducting additional interviews. Usually such people help us learn about interesting ways to use MailChimp products, and the time we devote to communicating with them pays off in the future.

Learn to move forward.

Of course, it is difficult. It is difficult to complete the study without resolving all the issues, analyzing each of the possible dependencies, trends, or clarifying other facts, but all this does not make you a better researcher, but only takes your time. If you find something interesting, but lying outside the framework of the current study, simply add another item to the list of questions you need to study and return to it later.

Do not rush to conclusions

Your job does not end with the classification of responses. , , 22% , , , . . ? , .

-: , , ( , ) .. .

? ? ? , . . , , , , .

, , . , .



. , .



, , . , - , , .

, . . .



, , – . .



. , . , , , .



. , - . , .

, . , , .

. , , -, .

, « », «-». , , .

Story

, , . . , , .

, , . , , . , , «». .

, - . , , , , . - , .

, , , . . , , .



, , , , . , . . – , , ?

. , , .

It may seem old-fashioned, but I always write a letter of thanks to all our respondents, leaving room for additional comments or questions. In the end, the “story” about the work of clients with our service is not limited to the scope of our interviews, and with luck, you can learn new details about their work with your product.

[ Translation of the 4th part of the book ]