Software Test Categories

The translation was made as a response to some comments on the translation Configuring the IDE to automatically run tests . After reading the article and looking at illustrative examples, you will be able to feel the difference between the various types of tests, which, in turn, will help you to write tests correctly and not to mix them in one heap. Every test is good at the right place at the right time!
- mazurov

You sometimes hear about small / medium / large / modular / integration / functional / scenario tests, but how many of us know what this means? In this article, my view on the types of tests.


')

Modular / Small

Let's start with the unit tests. The best definition I found is tests that run very quickly (less than 1 millisecond) and in the case when they do not pass, we do not need a debugger to identify the place where the error occurred. The definition itself makes a restriction on what the tests should be. For example, your tests should not perform any input / output (I / O) operations — this is one of the test execution conditions of less than 1 millisecond. The limit of 1 millisecond is very important because you want to run ALL (thousands) of your unit tests each time you change something, preferably every time you save the file . I can tolerate only 2 seconds and during those two seconds I want all my tests to be completed and nothing to break. It is very cool when you can just press ctrl + z several times to undo recent changes and fix the “red” test. Immediate feedback causes a habit like a drug addict. The absence of the need for debugger implies that the test area is localized (hence the name of the test - unit, a test of one class, for example).

The goal of unit tests is to test the conditional logic in your code — your ifs and loops. These are the places where most bugs are born (see bug theory ). That is why, if you are not doing other testing, unit tests are the best way to identify simple errors at the very beginning. If you have code passed through unit tests, then it will be easier to test at the higher levels described below.

KeyedMultiStackTest.java is an example of a good unit test from the Testability Browser . Feel how each test tells us a story. This is not testMethodA, testMethodB, ... - this is more like a script. Notice how the usual tests go first, to which you may have become accustomed, but toward the end of the file, the tests look a bit unfamiliar. This is explained by the fact that these are tests for borderline cases, which I discovered later. The funny thing about the test class in KeyMultiStack.java is that I rewrote it three times. I could not make it work on all tests. One of the tests always broke until I realized that the algorithm contained a significant drawback. By that time I had an almost working program and it was a key class for the bytecode analysis process. How will you feel when you have to bury something so fundamental in the system and rewrite it from scratch? It took me two days to recycle until all the tests passed successfully. After that, the entire application remains operational. This is what is called - "Aha! moment "- the moment when you realize how good the unit tests are.

Should each class have an appropriate unit test? In general, no. Many classes are not directly tested, but through testing something else. Usually simple value object objects have no unit tests. But do not confuse the lack of tests and incomplete test coverage. All classes / methods should be tested (test coverage). If you are test-infected (TDD), then it’s in your blood.

Medium / Functional

So, with the help of unit tests, you made sure that each class separately performs what you need. But how do you know that they will also work well together? To do this, we need to combine the related classes as it will be in the working version of the program and check the main ways of their interaction. The task is not to check if ify or cycles, but to control contracts between class interfaces. A good example of a functional test is MetricComputerTest.java . Notice that the input to each test is the internal class in the file being tested and the output is ClassConst.java. To get output, several classes interact with each other: parsing bytecode, analyzing execution paths and calculating the execution cost until the final cost is accepted.

Many of the classes are tested twice. Once using the unit tests described above, and the second time through functional tests. If you remove the unit tests, I’m still deeply confident that the functional tests will detect the changes that broke the code, but I won’t be sure which part of the code will have to crawl to fix the breakdown, as it can be in any class involved in the test. When the functional test fails, and the unit tests are performed successfully, I sometimes have to arm myself with a debager. After the problem place is discovered, I add a unit test that will help me 1) to prove that I understand what the bug is and 2) to prevent the bug from reappearing. Such unit tests explain the appearance of strange tests at the end of KeyedMultiStackTest.java . These are things that I did not think about when writing tests for the first time, but which needed to be turned on after many hours of debugging and finding the source of errors in the class from KeyedMultiStack.java .

Calculating metrics is only a small part of what the Testability Browser does. There are still functions that should be tested. You can think of functional tests as tests of related classes that form a single entity for the entire application. Here are some of these entities in the Browser: java bytecode parsing, java parsing, parsing with ++ code, ejection of metrics, 3 different types of reports and a tips engine. All of them correspond to a unique set of classes that interact with each other and need joint testing. Most often, these sets do not depend on each other.

Large / Acceptance / Scenary

What else do we have to test, other than ifs, loops, and interface interactions. There is another class of possible errors. You may incorrectly merge system components. For example, you can pass null instead of a report, incorrectly specify the path to the jar file for parsing, etc. These are not logical bugs, but binding bugs. Fortunately, binding bugs are easily reproducible and are usually accompanied by the appearance of an exception. An example of an acceptance test is TestabilityRunnerTest.java . See how the tests check the operation of the entire application and do not have too many assert checks. We don’t need a lot of them - we have already made sure that all the functions work and we only need to check those few places of the bundle of components.

---------------------------
translated.by/you/software-testing-categorization/into-ru/trans
(): Software Testing Categorization (http://googletesting.blogspot.com/2009/07/software-testing-categorization.html)
: © mazurov (Alexander MAZUROV).

UPD1 : New Translation from Testing Series: My Unified Bug Theory