How researchers at Uber apply and scale knowledge about human behavior

We have prepared for the readers of Habra a translation of the article of the Uber Labs team. Uber colleagues describe the work process of highly specialized type analysts (in the field of behavioral science) within a huge corporation, how their interaction with other types of analysts (UX researchers, product analysts) and colleagues from other teams (product, internal development), which they solve problems and how to approach them. Gleb Sologub, Skyeng analytics director, comments on the material.

At Uber Labs, we strive to use the ideas and methods of behavioral science to create intuitive and enjoyable programs and products. Members of our team have advanced degrees in psychology, marketing and cognitive sciences, have knowledge of subject areas such as decision making, motivation and training, methodological possibilities in the design of experiments, and are experts in statistical modeling and causal approaches. This knowledge allows us to deeply analyze the problems of increasing customer satisfaction, and thanks to our experience in methodology and statistics, we can measure the impact of business satisfaction on business (one of these approaches is the modeling of an intermediary ).

In this article, we will explain how our team applies theoretical knowledge of human behavior in practice, as well as how and why we work with product and marketing teams to improve the user experience of our customers. In particular, we look at an example with the recently launched Express POOL product.
')

Our path to behavioral science (about data)

In 2014, Uber was launched almost every day in a new city. Working groups in each region should have understood which communication strategies and products work best in the region, but most of them lacked experience in designing experiments and statistics. To solve this problem, we created Uber Labs - a team of researchers with a background in psychology, marketing and cognitive sciences. This centralized team had to use their abilities in the methodology and design of experiments and analyze the data through hierarchical modeling to improve our products for the benefit of passengers and drivers in different regions.

Individual consultations were effective, but we needed to scale this expertise to the ever-growing range of our products. By creating calculator templates for calculating sample size and statistical analysis using Shiny from RStudio, we provided non-technical teams with the opportunity to use our knowledge for their tasks. These tools, to work with which it was necessary to simply load their source data, included built-in checks for statistical assumptions and model fit, as well as automated selection of the appropriate analytical method for a particular data set. At the exit, the user received analysis results and clear explanations of these results. Later, together with the experiment platform development team, we created the process of analyzing and validating the data in our A / B testing tool . So it became even easier for other teams to efficiently analyze data.

As the company grew and expanded, creating new directions for product development, we realized that we could increase our influence by collaborating with the development teams directly. In early 2017, we began to apply applied behavioral knowledge in addition to statistics. We moved from a passive approach and support of already formulated ideas to an active one: we began to use our knowledge in the field of learning and memorization, which allowed us to propose concrete solutions based on existing scientific research. In addition to experiments, we began to support new directions: product strategy, program design, content optimization, and measurement of business impact.

Thanks to our training, behavioral experts are well versed in qualitative and quantitative research methods. Our field of activity has expanded, we have ceased to be just researchers, have become experts in data analysis and decided to focus on quantitative research methods as an important part of our work with data. In the UX department of Uber work highly qualified specialists who are engaged in qualitative research. By focusing on quantitative methods, such as testing theoretically sound ideas through experimentation and applying new statistical approaches, we complement the wider research ecosystem of Uber.

Our workflow: how we implement ideas and methods

We organized our workflow in such a way that we not only help solve problems by consulting at a particular moment, but also ensure long-term effectiveness by scaling knowledge and methods in the field of behavioral science using special templates and platforms. Tell us more about these processes.

1. Counseling is the most effective approach to solving tactical tasks at the level of a specific product or function. We work directly with product teams, marketing teams and other data experts and provide scientifically based advice on how to solve their problems.

2. In order to have a larger impact on the formation of product and analytical strategies, our team creates guidelines for content and development, as well as templates for R and Python, which allows our colleagues at Uber to independently study and reproduce our methods.

3. Finally, we work with teams throughout the company to provide access to our analytical tools and methodologies in one click. An example is our work with a team for developing a platform for experimentation on a tool for post- experimental analysis.

Our counseling is often associated with the application of theoretical knowledge to the problems that we describe in the example below. In our work, we apply a quantitative approach to solving such problems. All our work with data is built around questions about user behavior and is divided into three categories: a quantitative assessment of psychological constructs and processes, the use of behavioral science methods and the conduct of experimental analysis.

First, we use Uber data to quantify the hidden psychological constructs and processes that determine behavior. To do this, we either adapt the existing methods of social sciences and behavior, such as factor analysis , or develop new ones. To solve more difficult problems, we use some methods that are less commonly used in data science, for example, a mediator modeling approach developed by us or analysis of interrupted time series . Finally, we analyze data from various experiments, ranging from standard A / B tests to methods that are used when it is impossible or undesirable to conduct A / B tests, for example, experiments with randomized rewards .

In science, research is most often used for the further development of the theory, and not for solving applied problems. For our team, one of the most important aspects of the transition from theoretical knowledge to a specific business problem is the ability to apply applied research to improve user experience.

When we began to collaborate with product teams in the field of behavioral science, we were faced with the fact that even when concepts seem to be easy to understand and practical to use, their unsystematic use can lead to unintended consequences. Therefore, it is always necessary to take into account the situational and individual context . For example, in the science of behavior there is a phenomenon of loss aversion that is familiar to many. At first glance, its essence is obvious: people often prefer to avoid loss than to gain. However, there are many situations in which presenting something as a loss can upset or anger the user, rather than motivate him. For example, a long-time user of a loyalty program, for whom the entire experience of interacting with an application was based on getting points, can get angry if he is told that he will lose points if he does not make a purchase immediately. Even common trends such as loss aversion can have unintended or negative consequences if you work with them without regard to the context. No matter how potentially successful your approach is, we recommend conducting experiments to better understand and more accurately predict the result of its use.

Case: Express POOL

Since behavioral science is largely situational, most of our work is advising teams developing a specific product. Our collaboration with the Express POOL team is an example of how the applied behavioral science team applies theoretical research to product development.

In early 2018, Uber launched Express POOL . Like uberPOOL , Express POOL involves traveling together and sharing expenses with the passengers with whom you are on the way. Unlike uberPOOL, uberX and our other ridesharing products, when using Express POOL you have to wait a little longer for the appointment of a suitable car and walk to the specified landing site. Such changes allow you to create more direct and effective routes, which, in turn, makes the trip more accessible.

Passengers are used to the fact that the car quickly arrives exactly where they are, therefore, when developing the product, special attention was paid to how users interact with the new product. It became clear that many aspects need to be improved: customers canceled trips between the request and the selection of a suitable option. Passengers had to wait longer, and cancellations occurred more often than when using other products.

Usually we begin the process of consulting with a meeting with the team working on the product to understand the problem. This team includes a product manager, a marketing manager, a user experience researcher, an engineer, and a product data specialist. We review and take into account preliminary research team, such as the results of usability tests. In the case of Express Pool, having connected to the project, we learned the details described above.

After studying the context and understanding the general problem, we conducted a review of the special literature with an in-depth analysis of the available data of the behavioral science to determine the methodology for solving this problem. So, deeply plunging into the context, we transform our knowledge into real-world change scenarios for product teams and recommend ways to test these developments.

In this case, we began to study the literature on the science of behavior , to learn more about how people perceive time and expectation. We have identified three concepts that are important for understanding the waiting time: rejection of inaction , transparency of action, and the effect of the gradient of the goal . The concept of non-acceptance of inaction is obvious: people are afraid of inaction and want to be constantly busy. We also found that transparency of actions or disclosure to users of what happens to their request at any given moment, increases consumer evaluation of the product. Finally, the goal gradient effect is characterized by increasing motivation and great efforts that people are willing to apply when they feel that they are approaching their goal.

Considering this, we recommended showing progress while waiting, reflecting every step in the application, for example, specifying selected travel companions and notifying the customer about which machine was found.

Additional information, such as an explanation of the principle of calculating the time of arrival, can be obtained by clicking on the information icon. The Express POOL team checked these ideas with the help of A / B testing and recorded a decrease in the number of cancellations after calling the car by 11%.


Fig. 1. The test design of the Express POOL user interface shows detailed steps and uses icons to get additional information about the status of the order.

As described in this example, after a detailed study of the characteristics of human behavior, we have developed priority ideas based on assumptions about the potential impact and possible risks. To test the ideas we proposed, we organized and conducted experiments, and then analyzed the data obtained. The whole process of our research project, embodied in our work on Express POOL, is shown in Figure 2:


Figure 2. Usually our workflow begins with the formulation of the problem and ends with the conduct of experiments.

1. Problem definition
Get problem information from partner teams.
2. In-depth analysis and immersion in the science of behavior
Formulate the problem in relevant terms from the field of behavioral science.
3. Formulation of an idea based on available scientific knowledge.
Offer a specific product idea based on the results of a research study.
4. Prioritization
Together with other teams to prioritize ideas, taking into account the expected economic results and possible risks.
5. Experimentation
Conduct experiments to test ideas (develop variants of the experiment, determine the target audience, analyze the data, etc.).

Applying behavioral science helps increase product value.

Our work on Express POOL demonstrates the unique value that our research in the field of human behavioral characteristics, supported by decades of scientific experiments in this field, represent for the product in perspective. Armed with this information, we work together with UX-researchers and product analysts who use their skills to solve problems other than those we study. For example, during our experiment with Express POOL, product analysts carefully monitored the application’s metrics and found opportunities to improve the cancellation rate after a request. UX-researchers conducted test trips to understand the causes of passengers' difficulties and to understand the problem. As behavioral researchers, we used our knowledge and methodology to propose a solution that can be empirically tested.

We take into account our specialized set of skills and how we can increase the value of the product, when we choose, with which teams we cooperate and which projects we undertake. At the global level, we draw up a priority plan for the year, which is determined by the desired economic indicators of the product. At a more detailed level, the development team provides information on which areas of the product have the most pressing problems. Based on this, we choose which projects and in what sequence we will carry out together with other teams. It is important to note that our team considers these areas of development from the point of view of behavioral science, determining where it is worth using our applied knowledge and experience in quantitative analysis. In some cases this may mean excluding from the list of priority those experiments for which a strong theoretical base or qualitative research that does not require our methodological skills is not needed. We achieve serious results, always striving to exert maximum influence both on business and on the degree of relevance of the application of behavioral science.

Key findings

In the future, as Uber masters new development opportunities and improves existing products, we expect our team to have many opportunities to use behavioral science to offer our users the best service. In 2019, we will continue to collaborate with other teams on innovative and highly effective projects, and will also invest in scaling our knowledge to make behavioral science more accessible. We are pleased to continue to actively apply our theoretical and methodological knowledge and improve the efficiency of the functions, programs and platforms created in our company.

Comment from Gleb Sologub, Skyeng Analytics Director

At Skyeng, behavioral science methods are taken into account and used in the preparation of experiments and A / B tests on various landings, in the development of our mobile applications and the web platform for conducting lessons.

For example, through the A / B tests, we recently found out how the effects of priming affect the choice of package of lessons by our students and their decision to make a purchase depending on the location of the options on the payment page. Understanding motivation mechanisms helps us select the best motivational schemes for teachers and sales managers. And we embody knowledge in the field of teaching methodology in special interfaces that allow improving the efficiency of the teacher’s work.

I think there are not so many companies in the world that can afford to keep a separate team of behavioral analysts on staff. We at Skyeng are trying to train existing researchers so that they constantly expand their arsenal of methods and be able to choose the right ones for a specific task. And, by the way, our analytical team is growing - there are interesting vacancies !

Photo by meo from Pexels