Hello! My name is Evgeny Emelyanov, I am the project manager for the Mail.Ru Calendar project. Today I will tell you about how we have pumped the testing of mobile applications of the Calendar with the help of characters and scripts. Such testing is widely used in usability research and in studying user interaction with the interface. We decided to use similar techniques for the classic manual testing of mobile applications. At first, the team was skeptical, but the results were very positive, so we would like to share our experience with you.

Nota bene: in marketing and “user-centric” design, characters are fictional characters. They are different user groups, divided by geography, demography, behavior and habits. Marketers use characters to describe different market segments.

Characters can be useful in determining the goals, wishes, restrictions of consumers of a brand or product. They can help in making decisions about certain new developments, changes in the current functionality, design. User characters are a representation of the goals and behavior of hypothetical groups of users of a product. In most cases, characters are created according to data obtained from surveys and user interviews. The data contain descriptions of behavioral patterns, goals, skills, opportunities and the external environment. To recreate a more realistic image, fictional little personal traits are added. For each product, more than one character is usually created, but there must always be a main character representing the main target group.
')

The problem is overdue

Before we began to apply characters, the testing process in general terms was built as follows:

1. Developer builds build.
2. Smoke testing is performed.
3. The tasks performed in this build are checked.
4. The entire functional block affected by the changes is tested.
5. If the build can become a beta version, then full testing of all user cases is performed.

Check terms. In our case, “use case” is a small application usage scenario. For example, adding an event with certain parameters. Problems arose in two directions. First, yuzkessy were mostly "synthetic". For obvious reasons, testers contributed data that was cut off from reality, and more often they tried to purposefully “break” the application. It needs to be done, but still breaking something is a separate use case. Secondly, complex bugs appeared at the junction of junction cases or with a certain execution of several junction cases. Therefore, these bugs are more often found by users, not testers. This greatly saddened us.

Fictional friends

The project "Calendar" is small, cozy and almost family. We try to communicate with users directly, bypassing technical support. Therefore, we know our users well, their main tasks and are aware of most problems. Create portraits of major groups was not difficult.

Regular user : adds calendars with events (holidays, sports, movies), birthdays, uses various reminders about them. Occasionally adds personal events or new birthdays.



Mobile : uses the calendar only on a smartphone - iOS or Android. Very rarely comes into the web interface. Events are mostly personal, both one-time and periodical.



Active : uses the web interface and one of the mobile clients, iOS or Android, often switches between two clients. Creates and edits many events and tasks, often invites participants or is himself a participant in events.



Technocrat : uses the web interface and clients on both mobile platforms. Creates a lot of events and tasks. Uses non-standard approaches to tools, builds own schemes of work with events and tasks.



Drunken Master : a special kind, rare in nature, capable of wreaking havoc and destruction. Constantly confuses buttons, drives a stream of consciousness into forms, presses on Submit ten times, sends spam and tries in every way to break what it reaches.



These five user groups overlap user cases, but the scenarios are quite different (a set of user cases and the order of their execution). For example, fast and stable data synchronization between clients is critical for Active and Technocrat and is not particularly important for Mobile, since it uses only the mobile client. Based on these considerations, we have compiled usage scenarios for each character based on existing use cases.

Next, we put together several scenarios of interaction between the characters for group work in the calendar. An important point is that all scenarios, including group scenarios, were tested on the representatives of groups within the company. Thus, we checked at the same time the correctness of the choice of our characters, and the proximity of scenarios to real life. We used “corridor testing” on our colleagues, but for the purity of the experiment we also tried to survey third-party users using mail and instant messengers. “Corridor testing” turned out to be more effective, because when communicating with loyal users, a “help at any cost” corrective effect arose. And users were silent about some moments, adjusting to the expected, in their understanding, result.

The process has begun

Now the application testing process looks like this:

1. Developer builds build.
2. Smoke testing is performed.
3. The specific tasks in this build are checked.
4. Scenarios are tested for characters using the affected functional blocks.
5. If the build can become a beta, then all the scenarios of all the characters are tested in order of priority.

For example, here is one of our testing scenarios for the “Active” character. The script is designed to run within an hour:

• I view event notifications at the beginning of the working day in a mobile client (for some I review a detailed description);
• I transfer 1-2 events to another time;
• I add participants to one of the events in the web client;
• I create 1-2 events for the current day and 2-3 events for the current week;
• I create in the mobile client 2-3 tasks for an arbitrary date;
• I set the status "completed" to arbitrary tasks from the current list in the web client;
• in the offline mode in the mobile client, browse events, create a couple of new tasks, move one of the events, go online, check synchronization (we call this an imitation of travel in an elevator);
• I mark several tasks in the web client that I have completed, transfer 1-2 tasks to another day, create 2-3 events for an arbitrary date, shave an event from the Day in History calendar into a social network.

In order not to bore the details, the script is slightly shortened and simplified. As for the order of testing different characters and their scenarios, then, having statistics for different groups of users, we build priorities - the most numerous group is first tested. This is done in order to, first of all, stabilize the most demanded functionality and release alpha and beta versions more often.

Spoon of nuances

Naturally, the characters - this is not a pill for all diseases. There are problems associated with the use of methods, and problems that the characters do not solve.

The main one is episodic redundancy and the imposition of user cases for different characters and in different scenarios. We are struggling with this by breaking down execution cases by execution parameters. For example, we create events separately for the characters, and each has its own set of parameters, as close as possible to reality, covering all possible conditions for creating an event.

Another problem is that some user cases cannot be included in scripts, since they are either very rarely found in real life, or are synthetic for a character. We leave such use cases separately from the scenarios and pass them with full testing of all the functionality.

Profit

As a result of working with characters and scenarios, 3 updates of our mobile clients were released. One cannot refer to the improvement of statistics on finding offensive bugs by the users themselves, since the released versions differ greatly and comparing them with each other is not entirely correct. But there are other, equally important benefits.

First, testers are much more fun working with characters. The testing process has become more diverse, increased understanding of the product and the users themselves. Instead of hunting for abstract and hard-to-reproduce bugs in the real world, first of all, critical end-user problems are trapped.

Secondly, testers are also involved not only in functional, but also in usability testing. And now the number of opinions when discussing interfaces has been replenished with active users of the product with its own argument and vision.

Third, the entry threshold for new testers has decreased. Recently, we conducted an experiment and took a person without testing experience to the position of a tester. Apart from the “course of the young fighter” on general issues, effective testing began in about three days.

We must not forget that developers must also test their applications, and they have their own scripts for them. Niche products, like our Calendar, face the problem of using direct participants. Scenarios and characters perfectly cope with the question "why should I use it at all, I do not have such a need." In our case, the developers eventually begin to use the product in life, gradually moving away from the scenarios.

And the last - the characters are used by us not only in functional testing. Thanks to the work done, we have structured and brought to mind an important and useful tool that we can potentially use in marketing.

Yury Vetrov (@jvetrau), head of the Mail.Ru Group interface design team:

Characters are a powerful designer and designer tool that allows you to focus on the scenarios that are most in demand in real-life product use. It was great to see that it was used in the process of testing the quality of implementation, and not only with respect to usability.

In an ideal situation, bugs fix everything at once. But in life there are always a lot of tasks that push off the complete and irreversible bugfix - a new functionality, urgent hotfixes, etc. Therefore, we need a good way to prioritize both when fixing bugs and in finding them. To use for this the key scenarios of using the most important categories of users is the most. This means that, first of all, problems that prevent users most often are found and corrected.

Prior to that, they relied on characters for peer review of usability and user testing. Using them to verify the quality of implementation is an interesting and fairly fresh approach. I have been reading a lot about modern methods of working on interfaces for a long time and have never heard of it. So this is a great addition to the food team.