What is research testing?

And how does it differ from scenario testing (scenario testing)

This post is a translation of the article by James Bach. What is Exploratory Testing? This is the first translation from a series of articles by Bach about research testing and everything connected with it from the site http://www.satisfice.com . If you find an inaccuracy in the translation or an error in terminology, please report it in the comments to the article.


Exploratory testing is a powerful and enjoyable testing approach. In some cases, it may be more productive than routine scripting testing. I have not yet met a tester who would not apply research testing, at least at an unconscious level. However, few of us have studied this approach in detail, and it is not yet so recognized in our field. It is time for us to stop its denial, and publicly recognize the research approach as it is: scientific thinking in real time. Friends, this is a cool thing!

Parallel design and test execution

The simplest definition of research testing is to develop and run tests at the same time. Which is the opposite of the scenario approach (with its predefined testing procedures, whether manual or automated). Research tests, in contrast to scenario tests, are not defined in advance and are not performed in strict accordance with the plan. It sounds simple, but in practice everything is very vague. This is due to the fact that “defined” does not mean that we rigidly fix everything and everyone. Even if all test scenarios are carefully defined, then working with a large number of interesting details (for example, how to quickly type on the keyboard, or what types of program behavior are considered erroneous) remains at the discretion of the tester. In addition, even in a free-form search session, the test will include limitations on what part of the product to test or what strategy to use. A good research tester will record test ideas and use them in subsequent test cycles. Such notes are sometimes very similar to test scripts, even if they are not. Exploratory testing is sometimes confused with ad hoc testing. Ad hoc testing usually refers to the process of improvisation, searching for errors impromptu. By definition, anyone can do ad hoc testing. The term “research testing” (coined by Cem Kaner, in the book Testing Computer Software) means a thoughtful approach to ad hoc testing. Over the past ten years, James Whittaker, Sam Kaner, and I have been working to identify skills and techniques that make effective use of research testing. For example, research testing processes are fully defined and formulated, see the Microsoft's Windows 2000 Compatibility Certification program's General Functionality and Stability Test Procedure .

Balance between research and scenario testing

If each next test we perform is selected based on the results of the previous test, this means that we use research testing. We start searching and research, when we cannot say which tests should be performed, or when we have not had the opportunity These tests create, that is, the idea of ​​writing them did not even occur to us. If we go through the scenarios, and new information comes to light, which offers us the best testing strategy, we can go to the search mode (as in the case of finding a new error, which requires detailed consideration). On the other hand, we follow the scenario approach more when 1) the uncertainty in how we want to check is small 2) the new tests are relatively unimportant, 3) the need to ensure efficiency and reliability in performing these tests is worth the effort to work with such tests, 4 ) we are ready to pay for writing and maintaining tests.
')
The results of research testing are not necessarily radically different from those that we get through scenario testing, and both of these test approaches are fully compatible. Companies like Nortel and Microsoft typically use both approaches in the same project. However, there are many important differences between the two approaches.

Why do research testing?

The most discussed topics in managing an effective research testing cycle are testers, testing strategies, and reporting. The scenario-based approach to testing is an attempt to mechanize the testing process when an idea is taken from the head of a test designer and is presented on paper. This kind of testing is very useful. But testers using the research approach are of the opinion that recording test scripts and following them “dulls” the tester, making it difficult for him to quickly find key problems. The more intelligent we can do the testing, the more chances we will have that we test the application correctly and have time to do it in time. This is the power of research testing: the richness of this process is limited only by the breadth and depth of our imagination, as well as our understanding of the nature of the application under test.

Scenario testing occupy its niche. I can imagine testing situations where efficiency and reproducibility are so important that we have to write scripts for them or automate them. For example, in the case when the test platform is regularly unavailable, as in the case of client-server applications, in which there are only a few configured servers and they must be divided between the development and testing teams. Common sense tells us that we must thoroughly work out the test script in advance in order to get the most bang for your buck during the tests. Exploratory testing is especially useful in difficult testing situations when little is known about the product, or as part of preparing a test case set. The basic rule is this: exploratory testing is used in cases where the execution of the next test is not obvious, or when you want to go beyond the obvious. In my experience, this happens in most cases.