Kontur Mobile Test Session: 446 bugs in 5 hours

In December, Contour hosted the annual city test session of Yekaterinburg. At this time, 38 testers 5 hours looking for bugs in the new mobile application. Igor Borisykhin, testing specialist and organizer of the event, shared his experience, told what was new for the test session invented Contour and how to get to the event this year.

What is a test session

A test session is a competition for beginners and advanced testers and people who are not indifferent to testing, an analogue of a master class or a workshop for developers. At the test session, you can get acquainted with colleagues from other companies, check for a "strength" of a new product for yourself, find out who is better at finding bugs. Testing sessions are a traditional event for Yekaterinburg. What was the last fourth citywide test session can be found in the community of testers Yekaterinburg - UTC .

About format

The classic format of a test session involves testing a web service. The participant has a product analyst. There are N hours to search for bugs. At the end of the jury consider who found how many bugs. The best are prizes. To the classic format of the test session, Contour added:

Mobile app.
The product for testing was the mobile app Kontur.Conference for external and internal conferences. You can read more about the Kontur.Conference application in the markets: Google Play , App Store . Here's how it works:

MindMap instead of analytics.
We did not load participants with tons of boring analytics on the application; instead, each team had a map with the features of its work and a free field for research. For mappka used the program XMind .

Training.
According to the format of the test session in the process of searching for bugs, the participants studied new methods and techniques for testing mobile applications. More on items 2 and 3 a little further.

About the participants

The test session gathered 38 testers from different companies in Ekaterinburg: iRidium mobile, Ridero, Motorsport.com, Ural Airlines, Tochka, Advanta, Merkata, Extreme Pro, Flag Studio, BD Cube, ITM Holding, Digital Spectr, SkyDNS, Naumen and Contour .

Experience based testing

Instead of shipping testers with analytics, we suggested another way to get familiar with the application - testing based on experience.

Test based on experience - heuristic exploring in open multidimensional space. If it is simpler, research the application and draw conclusions correctly or not it works. Each participant in the test session has already been at conferences, so the application domain is familiar to him.

The basis of testing based on experience is based on three methods:

1. Error guessing - assume an error and check that it is not in the application.
2. Checklist-based testing - to come up with high-level checks for the application, and specific test cases to generate and check during testing.
3. Research testing - comparing the expected and obtained results of the behavior of the application based on their own experience.

But not all participants of the test session had experience in testing mobile applications.
In order to suggest research directions to beginners, share experiences and specific cases with experienced testers, during the test session we told the participants about mobile testing heuristics.

Mobile testing heuristics is a collection of research methods that contribute to the discovery of a previously unknown.

About pair testing

At the session, participants tested in pairs. 19 teams were compiled. Combined teams and balanced them according to the following criteria:

Experience in testing.
Newbies are more comfortable with newbies. Experienced with experienced. If a newbie works in a pair with a very experienced and experienced tester, in most cases, the experienced one dominates and prevents the novice colleagues from revealing themselves. If the experience of the participants is more than 4 years, for example, 5 and 10 years, then we can safely combine them into one team.

Experience in testing mobile applications.
A test session is an open event for testers with any specialization, so it is good if testers who are not only engaged in mobile testing fall into the same team.

Devices.
People came to the test session with their gadgets. We tried to ensure that the devices in the team were both iOS and Android.

A variety of communication.
We are for communication! Colleagues from the same company cannot be on the same team.

Pair testing helps to focus on the task, helps the tester keep moving while the other takes a break. The pair encourages each tester to explain and implement ideas. When a tester explains his thoughts to another person, the very process of formulation creates new ideas and cases. Pair testing is a great way to improve your communication skills and learn how to interact effectively with a colleague. For some, it was the first experience of testing in pairs, for someone - no. We hope that the participants will want to apply this method in their work.

About bugtracker

The bugtracking system was traditional for Contour Youtrack. Many testers worked with Youtrack for the first time, so we prepared a short video tutorial on how to work with the system and how to get bugs.

About the bug assessment system

According to the plan, 5 teams won, scoring more points. Let us give an example of a rating system.

In the bugtracker you could create two entities.
Task - suggestions for improving the system. Tasks were rated at 1 point.
Bug - defect in the product. Bugs had a different priority: crash, major, minor.

The crash priority bugs were infinite loading, freezing, data loss (data entered or edited is not saved), blocking the main functions of the device. Such bugs were estimated at 20 points.

The major ones included non-working buttons or links, non-editable fields, an unexpected result of execution, a layout that moved out (interfering with the work), a violation of data security. Such bugs were estimated at 10 points.

The minor ones included typos, disposed layout (not interfering with the work), incorrect animation, non-informative hints, unstable reproducing problem. Such bugs were estimated at 5 points.

About slang

Participants received features of slang for competent bug reports in mobile applications. A small example:

Tap - a brief touch on the touch screen, followed by removal.
Double-tap - two short finger touches, akin to a double-click.
Touch - touch longer than tap.
Touch and Hold - touch and hold. The touch is longer than the Touch.
Swipe (Slide) - a long slide with your finger across the screen.
Toast is a pop-up message on the surface of the application window.
Togl - state switch.
Title - title of the screen.
State - state, device orientation (portrait or landscape).

Now you, too, have remembered how tap differs from a swipe, togl from a toast, and a title from a state.

About prizes

According to the results of the test session, we selected 5 winning teams that scored the most points. Contour often holds competitions in sports programming, so we chose a proven scheme and present prizes like at ACM ICPC. On the common table with prizes, the team that scored the most points, the first chooses the prizes, the second for the team is the second, and so on.

Who tests the best

While the jury summed up, the participants could chat with colleagues, have a pizza bite, go on a tour of the office or spend time in the game zone: play a phone dart, 100 to 1 for testers, a prefix, a kicker.

In total, participants made 446 reports, of which 349 were of the bug type and 97 were of the task.

The jury accepted 278 and rejected 140 reports. They rejected the incomprehensibly described, non-reproducible or repeated within the same team bugs.

Of the accepted reports, 215 turned out to be bugs. Of these, 118 bugs have the priority of minor, 80 - major and 17 - crash.

I emphasize that this is statistics for all teams. In 215 reports there are a lot of repetitive between teams or device-dependent bugs. Therefore, the number of unique tasks that have migrated to the bugtracker of the application development team was 23.

All photos from the test session can be viewed here .

TestHackathonChallenge

We tried a new topic. And did a test session on mobile applications. The guys liked it, so we will continue in 2018, but we will add novelty. The next test session will be held at the end of the year, the announcement will appear on our blog - do not miss it!