Combinatorial testing: generation of test data and not only

Although the buzzword “pairwise” popularity is no longer the same, interviews are still asked about what this test-design technique is. However, not all testers (both those who come for an interview, and those who conduct it) can clearly formulate the answer to the question of why we need combinatorial techniques in general and pairwise in particular (the vast majority of errors are still on atomic values ​​of parameters and do not depend on others). The simple answer to this question, in my opinion, is to find bugs arising from explicit and implicit dependencies between parameters. For simple cases, the technique is unlikely to bring serious benefits, since they can be checked manually, and for a large number of parameters and complex dependencies between them, the number of tests is likely to be too large for manual testing. Therefore, the main use of combinatorial techniques (and, accordingly, tools that generate parameter combinations) is the automated generation of test data sets according to certain laws.

Most tools for generating combinatorial tests can produce a result in the form of a data file that can be input to the corresponding autotests. Such an example (using the PICT tool) and will be discussed below.

Example 1. Series and passport number

Provided that automated testing is used for a series and passport number, an exhaustive set of positive tests can be made, since the requirements for this field are strict - exactly two capital letters of the Ukrainian alphabet (except Ї,, b) and six digits from 0 to 9. In total, such tests will be (33-3) ² * 10 ⁶ = 9 * 10 ⁸ . However, rarely there are cases when the requirements for the field are so strict, and it is hardly necessary to carry out exhaustive testing. Most likely, it suffices to check the possibility of entering each individual letter and each individual digit at each position, respectively. The task of compiling such tests may well be solved by a combinatorial testing tool:

SERIES_1: ,,,,,,Є,,,І,,,,,,,,,,,,,,,,,,, SERIES_2: ,,,,,,Є,,,І,,,,,,,,,,,,,,,,,,, NUMBER_1: 0,1,2,3,4,5,6,7,8,9 NUMBER_2: 0,1,2,3,4,5,6,7,8,9 NUMBER_3: 0,1,2,3,4,5,6,7,8,9 NUMBER_4: 0,1,2,3,4,5,6,7,8,9 NUMBER_5: 0,1,2,3,4,5,6,7,8,9 NUMBER_6: 0,1,2,3,4,5,6,7,8,9 {SERIES_1, SERIES_2, NUMBER_1, NUMBER_2, NUMBER_3, NUMBER_4, NUMBER_5, NUMBER_6} @ 1 

Model 1

   4 6 3 1 1 5 І Є 8 3 8 9 9 3   3 0 5 8 6 2   4 3 4 1 3 1 Є  4 6 7 3 1 4   0 2 4 5 2 0 

Part of the simulation results

Similarly, you can create negative tests (PICT allows you to mark them with a special character "~").

 SERIES_1: ,,,,,,Є,,,І,,,,,,,,,,,,,,,,,,, SERIES_2: ,,,,,,Є,,,І,,,,,,,,,,,,,,,,,,, NUMBER_1: 0,1,2,3,4,5,6,7,8,9,~A,~B,~C,~D,~E,~F NUMBER_2: 0,1,2,3,4,5,6,7,8,9,~A,~B,~C,~D,~E,~F NUMBER_3: 0,1,2,3,4,5,6,7,8,9,~A,~B,~C,~D,~E,~F NUMBER_4: 0,1,2,3,4,5,6,7,8,9,~A,~B,~C,~D,~E,~F NUMBER_5: 0,1,2,3,4,5,6,7,8,9,~A,~B,~C,~D,~E,~F NUMBER_6: 0,1,2,3,4,5,6,7,8,9,~A,~B,~C,~D,~E,~F 

Model 2

   1 3 7 2 7 4   ~B 7 3 2 7 9  Є 8 8 2 0 ~A 8 

Part of the simulation results
')
From these files, you can then read line by line (the delimiter is a tab) and fill in the fields in the application.

Example 2. Increasing the test coverage using an additional parameter

Sometimes the bugs associated with validations depend on how the user enters invalid data: from the keyboard (physical or screen), using the context menu of copy-paste, hot keys, dragging the selected text. For example, often dragging and dropping text is not processed by client validation if incorrect data is entered in this way. The input method can be entered into the model as an additional parameter and taken into account when creating autotests.

 SERIES_1: ,,,,,,Є,,,І,,,,,,,,,,,,,,,,,,, SERIES_2: ,,,,,,Є,,,І,,,,,,,,,,,,,,,,,,, NUMBER_1: 0,1,2,3,4,5,6,7,8,9 NUMBER_2: 0,1,2,3,4,5,6,7,8,9 NUMBER_3: 0,1,2,3,4,5,6,7,8,9 NUMBER_4: 0,1,2,3,4,5,6,7,8,9 NUMBER_5: 0,1,2,3,4,5,6,7,8,9 NUMBER_6: 0,1,2,3,4,5,6,7,8,9 INPUT: keyboard, screen keys, context menu, copy paste, drag-n-drop 

Model 3

   0 8 0 8 5 9 keyboard   0 0 2 3 2 2 drag-n-drop   5 3 6 2 1 0 screen keys   3 9 4 1 6 7 context menu   9 9 0 7 4 4 copy paste 

Part of the simulation results

Example 3. Tests for decision-making systems, validation of requirements

For decision-making systems, comprehensive test suites are sometimes compiled, which can then be used not only for testing, but also for validating the requirements. Applying successively the rules of the system to each test, one can see whether inconsistent results are being obtained.

Validation of requirements is a very important part of testing in this case, because you can find hidden contradictions. The tool for generating combinatorial tests will not only create tests, but also specify the conditions imposed on the input data. If these conditions make any of the possible data unattainable, the tool will indicate this, which can serve as a signal for a thorough verification of the requirements for consistency.

 AGE: 0-17, 18-21, 22-65, >=66 CHILDREN: Y, N SMOKING: Y, N WORK: 0-5, 6-10, >=11 {AGE, CHILDREN, SMOKING, WORK} @ 4 IF [AGE] = "0-17" THEN [WORK] <> ">=11"; IF [WORK] =">=11" THEN [AGE] = "0-17"; 

Model 4

 Constraints Warning: Restrictive constraints. Output will not contain following values WORK: >=11 

Tool response to conflicting requirements

In this model, there are conflicting requirements that cut off the WORK value:> = 11, and it does not appear in any of the tests. Unfortunately, the tool does not give an answer to the question of which conditions cause a contradiction, it only shows which value is excluded from the tests. However, this information may be sufficient to isolate from the entire array of restrictions those that affect this parameter and analyze them for inconsistency.

An exhaustive set of tests can later be used for the cause-effect test design technique.

Example 4. Formation of environment parameters for configuration testing

Tools for combinatorial testing also allow you to make a list of possible configurations, which can then be sorted by popularity of use, cross out inappropriate, etc. If it is not necessary to carry out all the tests for each of the configurations, you can divide them evenly between the selected environments by adding the environment as another parameter to generate test data (as was done in the example with the data entry method).

 BROWSER: IE, Firefox, Chrome, Opera LANG: en, ru, ua OS: win, linux, android {BROWSER, LANG, OS} @ 1 IF [OS] = "linux" THEN [BROWSER] <> "IE"; 

Model 5

 IE ua win Firefox en win Opera ua linux Chrome ru android 

Simulation results

 SERIES_1: ,,,,,,Є,,,І,,,,,,,,,,,,,,,,,,, SERIES_2: ,,,,,,Є,,,І,,,,,,,,,,,,,,,,,,, NUMBER_1: 0,1,2,3,4,5,6,7,8,9 NUMBER_2: 0,1,2,3,4,5,6,7,8,9 NUMBER_3: 0,1,2,3,4,5,6,7,8,9 NUMBER_4: 0,1,2,3,4,5,6,7,8,9 NUMBER_5: 0,1,2,3,4,5,6,7,8,9 NUMBER_6: 0,1,2,3,4,5,6,7,8,9 ENVIRONMENT: IE ua win, Firefox en win, Opera ua linux, Chrome ru android 

Model 6

Example 5. Drawing up several tests with a large number of restrictions

Of course, combinatorial testing can be used to generate tests that are performed manually, but it seems to me that this should be done only if there are a very large number of restrictions that are difficult to keep in mind. Due to the presence of conditions, the number of tests may be limited, so to speak, in a natural way, and the tool will provide all possible test data that is suitable for all conditions imposed on them. In this case, the tests can be performed manually.

 AGE: 0-17, 18-21, 22-65, >=66 CHILDREN: 0, 1, 2, 3, 4, 5 SMOKING: Y, N WORK: 0-5, 6-10, >=11 IF [AGE] = "0-17" THEN [WORK] <> ">=11"; IF [AGE] = "0-17" THEN [CHILDREN] = 0; IF [AGE] = "18-21" THEN [CHILDREN] < 2; IF [CHILDREN] > 0 THEN [SMOKING] = "N"; IF [AGE] = ">=66" THEN [WORK] <> "0-5"; IF [AGE] = "0-17" OR [AGE] = "18-21" THEN [WORK] = "0-5"; 

Model 6

 22-65 2 N 0-5 18-21 1 N 0-5 >=66 2 N 6-10 22-65 4 N 6-10 22-65 5 N 6-10 22-65 3 N 6-10 >=66 4 N >=11 22-65 5 N >=11 0-17 0 Y 0-5 >=66 3 N >=11 22-65 4 N 0-5 22-65 2 N >=11 18-21 0 Y 0-5 22-65 0 Y >=11 22-65 1 N 6-10 22-65 3 N 0-5 >=66 1 N >=11 0-17 0 N 0-5 >=66 0 Y 6-10 >=66 5 N >=11 22-65 5 N 0-5 

Simulation results - 21 test