Testing of domains or more than analyzing boundary values

All testers have at least heard about such test design techniques as equivalence classes and analysis of boundary values. It would seem that it could be simpler: select classes, take one value in each, check the class boundaries and values ​​to the left and right of the boundaries. But is it always so simple? What if, after partitioning into classes, it turns out that there is no problem with borders, they cannot be defined, since the data cannot be ordered? What if the tested parameters are interconnected by some logic and depend on each other? How many tests are enough? Below will be considered the possibilities of the two main techniques of test design, exceeding those laid down in their direct definition.

Domain - a mathematical term - a collection of all possible values ​​of a variable. Testing of definition domains regards a program as a function of many variables, each of which takes a finite set of values. Each such set can be divided into at least two equivalence classes - valid and non-valid values.

Testing definition domains involves three steps:

1. the selection of subregions for each parameter, all elements of which presumably lead to the same program behavior (to reduce the number of tests);
2. selection of specific values ​​for testing inside each class (including for detecting errors related to the fact that the definition domain is set incorrectly);
3. a combination of these values ​​(to increase test coverage and to detect errors depending on the interaction of several parameters).

Experienced (and not so) testers are now skeptical that the tasks above are solved by three test design techniques — a division into equivalence classes, an analysis of boundary values ​​and a pair-wise search. This is true, because then we will discuss some features of the use of these techniques, which will help to detect more errors and / or reduce testing time, while preserving the maximum level of test coverage, and, therefore, confidence in the quality of the program.

Equivalence classes

A few simple rules.

1. If the domain of definition of a parameter is a range, then it makes sense to distinguish three equivalence classes: to the left of the range (invalid values), the range itself (valid values) and to the right of the range (again invalid). When allocating classes, it is necessary to use inclusive boundaries for the purpose of unambiguity and accuracy: the same value cannot belong to two classes simultaneously.
2. If the domain of definition is a set of disordered data, then at least two classes can always be distinguished - valid and non-valid values. The resulting partition can be "split" further. For example, a set of Latin letters can be divided into two subsets: Latin in upper and lower case, respectively.

The example above uses the obvious way of splitting into subclasses, but it is not the only one. Such a partition is not always advisable in the sense that it does not so often contain errors. Here are some other tricks:

• in terms of frequency of use by end users (for example, for parameters such as login and password, it may make sense to single out the character class “qwertyQWERTY1234567980” into a separate class);
• random subclasses of equal size (providing conditional test coverage if there are no other logical ways to subclass).

There are linear (ordered) and nonlinear (unordered) equivalence classes. Obviously, it is impossible to apply the analysis of boundary values ​​to the latter, i.e. There is no logical way to select elements that are more likely to result in an error. An example of such a class would be a variety of special characters that can be entered from the keyboard. For crushing such a class, a second method may be useful. The input parameter for its application will be the number of subclasses that are planned to be used in testing: one will be taken from each one.
')
Typical errors of this stage of testing domains: too many or too few classes, classes are allocated incorrectly (in relation to the functionality of the program).

Selection of values

After the division into equivalence classes is completed, it is necessary to select values ​​from each class that will be used in the tests. The analysis of boundary values ​​is only one of the methods, and is suitable only for linear classes. What to do in other cases?

1. Random selection. It is highly desirable that with each subsequent test run, a different value from the class will be chosen at random. In this case, you can use a random selection with a return (the value selected earlier can be selected again with the same probability) or without return (the value selected earlier can no longer be selected, thereby increasing the probability of selecting the values ​​remaining in the class).
2. Proportional partitioning. There are different algorithms whose main purpose is to reduce the risk of incorrect division into equivalence classes. To do this, you can take several values ​​from each class (the number increases with the utility of a particular function of the program). Another way is to take from each class not a fixed number of values, but a fixed part of the class. Thus, for classes containing more elements, more tests will result.
3. Analysis of boundary values. It should be remembered that the main idea of ​​this technique is the selection of values ​​that lead to errors more likely than others. This technique is not limited directly to the controls on the program screen. In addition to numeric range boundaries, it is worth remembering time limits (for example, the period of free use of the program), cycle boundaries (number of incorrect password entries), type boundaries (even if according to the specification you can enter an integer number that is not limited from above, this number is or else it will be limited by the maximum value of the integer data type that the programmer chose in implementing this function). There are other boundaries associated with non-functional types of testing - performance, configurations.
4. Empirical knowledge. Some values ​​may be chosen more often than others or may be of particular use in terms of the business logic of the application. The ideas of such tests can be prompted by a person who is well-versed in the subject area of ​​the program.

When analyzing the boundary values ​​and allocating equivalence classes for numerical parameters, special attention should be paid to the result of the calculations. Such a partitioning and selection of values ​​for testing will help to find important errors. What restrictions are imposed on the range of values, i.e. the result of the calculations, and what values ​​should the input parameters take? What you need to set the input values ​​to go beyond this area? For example, if the result of the calculation should be positive, it is worth to distinguish three equivalence classes and the corresponding boundary values:

• input data for which the result is strictly positive (valid class);
• input data for which the result is zero (invalid class);
• input data for which the result is negative (invalid class).

It may be that the values ​​that appeared during such a partition in non-valid classes are allowed for input according to the specification. This may be an error in writing requirements that business analysts should point out.

Combinations of values

Defects that depend on the input data can be divided into those that occur with a specific value of one parameter, and those for the occurrence of which you need a combination of specific values ​​of more than one parameter. For the detection of the latter, combinatorial testing techniques are used, one of which is pairwise testing (pairwise).

Combinatorial techniques can be applied when equivalence classes are allocated for each parameter, and values ​​are chosen for which tests will be conducted for each parameter separately. You can use several strategies for making combinations:

• "Weak" vs. “Strong” combination - a weak one will make it possible to compose a minimum of tests for the detection of all defects occurring on a specific value of one parameter; a strong one is intended to detect defects that occur at the “junction” of parameter values;
• "Normal" vs. “Reliable” combination - normal uses only valid values ​​selected for testing, reliable - everything.

These are independent complementary characteristics, i.e., for example, a weak normal combination can be applied. Such a strategy involves the compilation of tests in which all the valid values ​​of each parameter selected for testing are met at least once.

Example:

• the parameter A can take valid values ​​a1, a2, a3 and non-valid a_4, a_5;
• parameter B can take the valid value b1 and non-valid b_2, b_3, b_4;
• parameter C can take valid values ​​c1, c2, c3 and c4 and invalid c_1.

Weak normal combination will give the following tests:

• a1, b1, c1
• a2, b1, c2
• a3, b1, c3
• a1, b1, c4

Strong normal combination - all possible combinations of valid values ​​of each of the parameters:

• a1, b1, c1
• a1, b1, c2
• a1, b1, c3
• a1, b1, c4
• a2, b1, c1
• a2, b1, c2
• a2, b1, c3
• a2, b1, c4
• a3, b1, c1
• a3, b1, c2
• a3, b1, c3
• a3, b1, c4

Similarly, you can create test suites using the strategies of weak reliable and strong reliable combination (try this for the example above).

Obviously, when using a strong and / or reliable combination, the number of tests will increase dramatically with an increase in the number of values ​​of any of the parameters and, of course, with an increase in the number of parameters themselves. The pairwise search technique (pairwise) is one of the ways to reduce the number of tests, while trying to preserve the quality of testing, i.e. minimize the number of undetected errors. But using this technique, it is important to understand that errors at the junction of more than two parameter values ​​will remain undiscovered.

Another way to reduce the number of tests is to find out if there are dependencies between the input parameters, and take this into account in the tests. This is probably not always: often testing the black box does not allow "to look inside." In this case, you can try to identify dependencies empirically and take them into account in combinatorial tests. However, there is a risk of making a mistake in identifying such patterns.

Combinatorial tests can and should be compiled using appropriate tools to avoid the human factor.

I sincerely hope that the above will help you design effective tests.