Exploratory testing: when to use it and how to do it

Many are skeptical of research testing, as they believe that this is a waste of time and resources. But actually it is not. In this article I will tell you when research testing will benefit the project. In the Russian-language literature there are many different definitions for the term “research testing”. Often, this concept implies ad-hoc testing and vice versa. Why it happened so historically can be found there - Research Testing 3.0 . To avoid confusion when reading an article, check the clock and fix the definitions.

What is research testing

Ad-hoc testing
By ad-hoc testing, we mean testing without the use of specifications, plans and test cases developed: pure improvisation.
')
Research testing
A more formal version of ad-hoc: testing that does not require writing test cases, but implying that each subsequent test is selected based on the result of the previous test. And according to Sam Kaner, “Testing Computer Software”, “research testing” is a thoughtful approach to ad-hoc testing.

Scenario testing
Classic testing of pre-written and documented scenarios.

In favor of scenario testing:

• comparative ease of planning: test cases can be easily divided between different testers or teams.
• important cases will not go unfulfilled;
• it is easier to estimate the percentage of project coverage by testing and to understand which part has already been tested;
• it is easier to introduce a new person into the project: the actions that are expected of him are already structured in the sequence of steps of test scenarios;
• with a sufficiently detailed description of test scenarios, the tester's qualifications may be minimal;
• developed test scenarios can be transferred to the customer for product acceptance testing.

In favor of research testing:

• without predictability and rigid attachment to a fixed sequence of steps, more defects can be found. Basically it will be defects that are not related to the main functionality;
• no need to spend time on a thorough thorough description of all scenarios;
• do not need support for test scripts;
• there is no getting used to the test scenarios, and their passage does not occur "without looking";
• not lost whole vision of the product;
• critical defects are faster;
• test speed increases;
• You can immediately begin to test the product, even if there are no requirements at all. Besides the fact that it is fun, it is also a significant saving of time, in comparison with a separate study of documentation and subsequent testing;
• more interesting and creative. Tests are limited only by the imagination of the person passing by and his depth of knowledge about the product.

Re-read these points again, but with the thought of why the pros of scenario testing may be a minus for research and vice versa.

When can you apply research testing in its pure form

Little time
If the test documentation is written, but there is already no time to pass the tests, you need to select the most critical areas of the application that you can actually test for the time available. Create a checklist of ideas and test around them.

Difficulties with requirements
No requirements, they are not complete or outdated and there is no possibility to actualize them.

Small project
The product is small, and developing test scripts will take longer than the testing process itself.
When can research testing be used in addition to routine testing?

Testers constantly pass the same test cases.
With repeated passing of the same tests, for example, during regression testing, testers lose concentration and begin to miss defects. In this case, research testing helps to look at the project from a new angle and find the missing defects.
The tester is distracted from the template actions and feels more of an ordinary user. This helps to find defects that have a greater effect on the end user of the product being developed.
Here you can take advantage of the concept of tours. Read more in Russian - Life is movement! And testing is life :) Most of the tours are used by testers intuitively, and the rest do not bring much benefit, but the fighting spirit and the desire to explore after reading the article should appear for sure.

A sudden change request came
There is no time to develop new scenarios, as everyone is busy with other planned tasks or changes will require reworking most of the documentation. In this situation, testing by the research method may be the most optimal.

When you want to be safe
The product has already been tested on scripts, but still I want to make sure that nothing was missed.

When research testing alone is not enough:

Standardized application
Applications that work by standards and guests, as well as systems for which the slightest deviation may be critical. These may be applications responsible for flying missiles or conducting financial transactions.

Integration testing is underway.
In this case, exploratory testing is possible, for example, when testing an API. But usually integration testing is carried out to check the interaction of internal components of applications. This work is well documented and often automated.

Test scripts are outsourced
Outsourcing outsourcing is different, but it is easier to control the task and the percentage of its implementation according to formalized scenarios.

Long project
Testers can be connected to the project for a certain phase, and then, while the developers implement the new functionality, they can work on other projects. If you do not test a specific functionality for a long time, then its specificity is forgotten.

Dispelling myths or how to apply research testing

Myth 1:

“Research testing cannot be monitored, it cannot be managed. It's hard to tell when it's time to stop and if all the functionality is covered. ”

Sometimes research testing is perceived as an antonym to the scenario and is regarded as testing in complete chaos.
In fact, the effect of measurability and parallelization of tasks is quite simple to achieve. Enough to fix the amount of work and divide it into time-measurable parts.

Myth 2:

"You can not entrust the test to the first comer"

This is partly true. But scenario testing should not be given to a “random” person. In practice, it is impossible to test the product well, following only the previously prepared steps. There is always a desire to deviate from carefully considered scenarios and work with the details - add negative checks, check work with interruptions, and so on. And this is good, since it is impossible to cover the product with 100% tests and it is never possible to completely exclude the human error factor.
In general, improving the skills of a QA team is always one of the goals of a QA unit. Using research testing, engineers use the intuition and experience accumulated earlier and get used to constantly analyzing the product.

Myth 3:

“It is difficult to“ sell ”research testing to a customer, to explain his need”

In fact, for the customer, the result and transparency of the processes is important. In this case, the result is a product that satisfies the customer's ideas about quality. And the necessary transparency of the processes can be achieved with the help of competent reports.

If, in the case of scenario testing, a simplified report can be a list of test scenarios with the result put, then a slightly different format should be developed for the research testing report.
A “good” research testing report might look like this:

• the list of tested product functionalities (to roughly evaluate the test coverage, as well as the need for additional research);
• list of defects (found in general or only the most critical - depending on for whom and at what stage of testing a report is made. And also depending on the total number of defects in the product as a whole);
• internal reports can be supplemented with problems, questions and / or observations;
• risks. Here it is important to talk about what has not been tested and why it happened - the functionality was not part of the scoop, the server was not working, there was no suitable test data, and so on;
• a brief conclusion on the results of testing (depending on the initial purpose of testing - for example, whether it is possible to transfer the product to the customer for review).

Naturally, these items do not lose relevance for testing reports by other methods.

findings

Research testing does not mean a complete lack of documentation and chaos, but is a powerful tool.
Using the ranking of types of testing from fully research to fully scenario, detailing the structured checklists, you can choose the optimal level of documentation for your project and save time.
Scenario and exploratory testing are fully compatible and compensate for each other’s shortcomings. You can cover the complex technical aspects of the project with detailed tests and write surface checklists for the user interface.
Be flexible. Develop a strategy that best suits your product. Quality projects to you.