UX research and evidence strength

Hello to all! I bring to your attention the translation of the article Dr. Philip Hodgson ( @bpusability on Twitter). He has a BSc, MA and PhD in experimental psychology. He has more than 20 years of experience as a researcher, consultant and trainer in usability, user experience, human factors and experimental psychology. His work has influenced product and system design in the areas of consumption, telecommunications, manufacturing, packaging, public safety, web and medical areas for the markets of North America, Europe and Asia.

The concept of "strength of proof" plays an important role in all areas of research, but is rarely discussed in the context of user interaction research. We will find out what this means in a UX study, and group the research methods by the power of the data they provide.

Someone once said: "There are no questions in usability." I think it was me. I admit that it was not the best statement. It does not stand next to words such as “Never on the margins of human conflicts” and “One small step for a man”, but, nevertheless, it makes sense and leads to a useful rule of thumb for UX researchers.

Let me explain.

A few years ago, while working at a large corporation and preparing a usability test, the project manager called me and asked me to send him a list of questions about usability.

“There are no questions in usability,” I replied.

“What do you mean?” He asked, “How can there be no questions? How do you plan to understand whether people like our new design? ".

“And I'm not trying to understand whether he likes them or not,” I grumbled. “I'm trying to figure out if they can use it. I have a list of tasks, not questions. ”

Good and bad data

Requests to add explicit questions like “what do you think?” In the UX study not only indicates that some stakeholders do not understand the purpose of usability testing, but also that they believe in the great value of each answer of the testing participant. This shows that they are not aware of the concept of good and bad data and, as a result, they believe that everything the user has said is useful.

But it is not.

There are strong and weak data. This is true for all areas of scientific research, be it the development of a new drug, the discovery of a new planet, the discovery of a crime, or the evaluation of a new software interface.

UX research is the direct observation of what people do. This is not a collection of their opinions. This is because as data, opinions are useless . For every 10 people who like your design, there are 10 others who will hate it, and another 10 who will not care at all. Opinions are not proof.

Behavior, on the contrary, is evidence. That is why the detective will rather catch someone by the hand at the time of the commission of the crime than simply believe someone for the word. Therefore, the advice is often repeated: “Pay attention to what people do, and not what they say . ” This advice has almost become a cliché in UX, but with this you can start a discussion about something important, for example, about the strength of evidence. It is a good idea that some data is supported by strong evidence, some relatively strong, and some weak. Nobody wants weak evidence at the heart of their product development.

Evidence in UX research

Evidence is what we use to back up our statements and arguments. This is what gives us credibility when we make decisions on specific design parameters, product features, on when to complete the next iteration in design, on “do / not do” decisions and on launching a new product, service or site. Proof is what we provide to our development team and what we lay on the table during disagreements and disputes. We substantiate our arguments with evidence based on good data. Data is part of the study. "Data! Data! Data! ”Shouted Sherlock Holmes. "I can't make bricks without clay."

It may seem that UX studies are events conducted on the principle of “first method” (“we need a usability test”, “I want to conduct a contextual survey”, “let's fix card sorting”), but a UX researcher that focuses on primary research questions , operates on the principle of "first data": "What type of data should I collect in order to provide reliable and convincing evidence on this issue?" . And then follows the method.

What is strong evidence?

Strong evidence follows from data that is reliable and reliable.

In usability testing, reliable data is things like an indicator of task completion and efficiency, but not aesthetic appeal or personal preference.

Reliable data is data from a survey that was conducted again using the same method as last time, but with the participation of other respondents.

Regardless of the method, research data must be reliable and reliable. Otherwise, they are simply thrown away.

In UX studies, strong data comes from the performance of tasks, from observations of the user (objective and independent ), and when the user is unbiasedly caught “by the hand”. The data becomes strong along with our level of confidence and assures us that the continuation of the study is unlikely to change our level of confidence in our findings.

Below is a brief systematization of methods based on levels of evidence. In essence, this is a systematization of the data types that methods provide. It is assumed that each method was well thought out. This is not an exhaustive list, but it does include a list of the main methods that UX researchers typically use in creating user-oriented design.

Examples of strong UX evidence

Strong UX-proofs inevitably include target users performing tasks, or involved in activity that is relevant to the concept being developed or the problem under investigation. Here they are:

• Contextual research (field ethnographic research or other options, when the behavior of users is recorded when they perform their work or achieve their goals)
• Forming and summing usability tests in which users perform tasks in the product interface
• Web / search analytics or any type of automatically collected product usage data
• A / B or multivariate testing
• Controlled experiments
• Task analysis
• Secondary behavioral research based on meta-analyzes and peer-reviewed articles, as well as previous UX reports that fully describe the method used

Examples of relatively strong UX evidence

In order to define this category, the data must be based on the results of research, which at a minimum include the performance of tasks - either by users or experts, or include independent reporting of actual behavior. These methods are often the precursors of methods from the “strong” category. They fall into this category because the data they give is usually less constant and less accurate.

• Heuristic estimates
• Cognitive step by step problem solving
• Feedback usability experts who performed real tasks
• Interview or any kind of report on your own behavior of the Jobs to be done type
• Building user routes
• Diary studies
• Card sorting
• Eye tracking
• Pop-up study "partisans" in cafes, libraries, etc.

Examples of weak UX evidence

Decisions based on weak or erroneous data can cost companies millions of dollars if these decisions translate into poor design, poor marketing, or incorrect statements about the product.

An obvious question arises: why do research, the result of which is bad data?

There is no need.

Data from these methods are not needed in UX studies. They are only slightly better than simple assumptions. If you can choose to spend a project budget on such methods or on charity, select the latter.

• Any type of pseudo-usability testing, for example, interviewing people about which design they like more, or tests that largely depend on interviews for primary data collection
• Undefined testing out loud, which allows users to think that they are experts, not respondents who perform tasks
• Evaluation of usability, even by experts. This is the same as “kicking tires.”
• Group focus ( don't make me start )
• Polls (you have the right to disagree, but only if you slept during the US elections in 2016)
• Intuition, appeal to the authorities or personal experience
• Opinion of friends, colleagues, family, your boss, managers and company executives

How to evaluate the power of evidence in a study or report

Start asking these questions:

• Why should I believe this statement?
• How good is the proof?
• Can I count on these findings?

There are no dirty tricks in these questions: anyone who presents the findings of the study should be able to answer them.

During the study, ask yourself:

• I observe people who work (perform tasks in the prototype) or listen to what they say (share opinions on design)?
• Interviewed people reflect on what they can do in the future, or rely on events that happened to them in the past?

Some time ago I created a checklist for evaluating research methods . If you want to give the research a good shake, you will find a lot of interesting things there.

I started this article by promising a rule of thumb. Here it is. Use it as a mantra when evaluating the power of user research:

Behavior - strong data. Opinions - weak data.