Crowd testing service Utest: how to max out

Recently, many services have appeared on the market, offering for the monetary reward their services to ensure the high quality of your product. More precisely, to control this very quality. However, the beautiful sign often hides the problems that negate all the benefits of such “testing by the crowd”. We want to share our experience and save you from repeating some mistakes.

KamaGames Studio in its work tried to cooperate with several such platforms. The most long-term cooperation was with Utest . By the way, we recently wrote that the guys raised substantial investments in the amount of $ 43 million. We have been a subscriber of one of the major packages of services throughout the year. Therefore, we can judge from personal experience about things that are required to create the most profitable relationship with a platform for crowd testing.

Tips for organizing teams

Utest can be used for different tasks solved by QA. Note that in all cases we did not rely solely on the external platform. Always with them worked a highly qualified QA-department within the company. What types of load can be delegated to Utest and others like it?
')

1. Interaction group “Smok-test”

Daily check service health of the vital elements of your application, which is in the publication, on the checklists. As a result of using the Smok-test group, the service regularly provides a litmus test of the health of your application (availability for download, correct installation, launch, operation of a store, etc., etc.) and in case of problems, sends alerts with a notification to mailbox. Simple and clear work.
Efficiency: 7 out of 10.

2. Interaction group “Implementation of business requirements”

Verification of the implementation of regulated business requirements for which the test design was written (performed iteratively). The service in this case is the strength of the “manual” tester who performs the verification of the implementation of the functionality planned for this iteration.
Efficiency: 7 out of 10.

3. Interaction group “Unified Cases”

Verification of compliance of the application with your corporate quality standards (performed twice (with alpha and beta) + possible regressions). As a result of using the “Unified Cases” group, the service is the strength of a “manual” tester who checks the application's compliance with your unified corporate quality standards (non-functional tests: the correctness of an incoming call processing by an application, the correctness of a transfer application from / to a card memory, etc., etc.)
Efficiency: 7 out of 10.

4. Interaction group “Research”

Free floating ad hoc cases. It is carried out in parallel with the interaction group “Implementation of business requirements”. As a result of the use of the Research group, the service represents the strength of the “manual” beta tester describing his impressions and problems in areas not regulated by test design. Efficiency: 5 out of 10.

5. Interaction group “Usability guarantee”

Verification of compliance of the application with international standards of usability (performed twice (with alpha and beta) + regressions are possible). As a result of using the “Usability Assurance” group, the service is an alternative to the work of a usability tester, who performs a reasoned verification of the application's compliance with international quality standards of usability. To listen to their advice or not is your right.
Efficiency: 5 out of 10.

6. Group "Guaranteed server performance under load" (very difficult for communication group)

Load tests on your server plus verification of the implementation of the server protocol. As a result of the use of the group, the service is an alternative (or addition) to the work of the tester of the automation engineer who is performing the implementation of load testing scripts (checking the implementation of the protocol) from his site at the appointed time.
Efficiency: 4 out of 10.

7. Group “Localization check”

Validation of localization. The service does not have any pronounced features of nationality and religion. The service here provides a “multiplied” “manual” tester, a native speaker and culture necessary to verify localization.
Efficiency: 8 out of 10.

8. Group “Compliance with progress”

Verifying the functionality of a published application on newly released devices and operating systems. The work of a “manual” tester, checking the performance of a published application on a new device. You are always aware of the relevance of your application to progress, even if you do not have the corresponding device.
Efficiency: 8 out of 10.

General tips on the features of the interaction and "rake"

Above, we told you why you can use Utest and similar services. However, the path to the goals will be thorny: below are simple but important tips on how to avoid the failing in your work.

1. All interaction takes place in English, it is necessary to take this into account when developing test-design documentation and hiring testers who will work with Utest in the state.

2. Communication and management takes almost all the working time of a full-time QA-manager: you need to constantly ping the service manager and keep track of what is happening, because The service manager has relationships with other customers, rarely, but incidents happen. Most often due to the fact that the parties did not understand each other. Sometimes it is useful for bargaining and for pressure.

Real examples from practice:

• The service manager at the same time took four assemblies of various applications into operation (for the “Research” interaction group). A day later, we do a control ping, and it turns out that testing was started only in three assemblies (he launched four, but did not set deadlines and tasks - “day for nothing”);
• The service manager took to work one assembly of the application (by interaction group “Research”). After a day we discover that the service (here and everywhere in the text, the term “service” means all platform participants who took part in testing our products) did not reveal a single problem, which is strange for the alpha version. After “pinging”, the service manager sent a “magic” message to testers, as a result of which “useful” problems were identified.

3. The interaction takes place through the system bug / task-tracking service, without the possibility of integration into your system.

In the service tracking, a lot of functionalities are implemented (a single database of registration / authorization of testers, evaluation by QA-manager of problems found, various analytics, task / bug-turnover, etc., etc.), which does not allow integrating it into your system bug / Task-tracking, so a lot of time is spent on copy-paste problems and tasks. We did not manage to organize the import to Redmine.

4. The internal QA-manager needs to read every problem brought by the testers of the service in the tracker, and reproduce it.

The participants of the Utest testing community after evaluating the criticality of the identified problem by the QA-manager receive a cash reward from the service. In this connection, it is rare, but still not quite honest testers who try to deceive both Utest and you.

5. It is necessary to form requirements for the description of the problem by the testers of the service.

You must clearly formulate the service manager the minimum information you need to describe the problem (device, operating system, reproducibility (how often it is played on which devices), playback steps, actual and expected results, logs, video process recording, etc., etc. .) and require compliance with this minimum. Otherwise, get a bunch of useless information.

6. It is necessary to develop a template for setting the task to the Utest manager.

This is extremely important to save time and eliminate human “forgotten”. The template should include a minimum of necessary conditions, for example, consist of the following items:

• Application name - Project;
• Build No.;
• Interaction group (see points 1);
• Task body (main text, features);
• Coating devices;
• Coating operating systems;
• The necessary deadlines for the task;
• Comments;
• Link to the storage location of the assembly;
• Reference to the storage of regulatory documentation
• Etc. etc.

findings

Using Utest and similar services can certainly be useful. For us, this benefit consisted in expanding “human resources” at any given time and additional testing of products both at the development stage and at the operation stage.

Nevertheless, the service will never replace the homebrew “fat” (in the best sense) tester:

• In terms of the level of knowledge and understanding of the specifics of your product, a regular tester will always be stronger on the head, if only because service testers are coming and going (often a one-time phenomenon);
• A staff member’s motivation is higher because for him it is a product in which he put a lot of strength and energy;
• The level of theoretical and practical knowledge in testing your tester is obvious to you, but the tester of the service is not;
• Monitoring the progress of the task (hints, degree of completeness, coverage) of the staff tester is carried out as efficiently as possible, and the progress of the task execution by the service tester is less effective.

Use a similar service as an additional tool to check the quality of the “home-grown” staff - even if this staff is yourself in your indie team. Be prepared to spend a large amount of time and effort on communication, and the service will surely reciprocate in the form of an additional quality indicator. And, of course, look at the prices. The site Utest presents a calculator of the cost of services , which means not only the opacity of the price, but also the possibility of some bargaining.

The author of the text: Yuri Golovin, head of QA KamaGames Studio.