Principles of software testing. Personal translation from the book “The Art of Testing” by G. Myers
Continuing to pay tribute to the issues of psychology in the process of testing, we can define a set of vital (read, damn vital) principles of testing. Many of them will seem obvious, which, however, does not often prevent them from being neglected. Here they are, and then - their detailed consideration:
1. Mandatory part of testing - determining the expected result;
2. Programmers should avoid testing their own programs (and code sections);
3. Organizations that create programs should avoid testing their own programs;
4. The testing process should include a thorough check of the results of each test;
5. Test cases should be designed for both correct and expected input conditions, as well as for incorrect and unexpected;
6. The study of the System for the fact that it does not do what it should be - only half the story. The second part is to figure out what she is doing wrong;
7. Avoid one-time test cases only if the program itself is not one-time. Disposable test cases for disposable programs. In other cases, those should be avoided;
8. Do not engage in the testing process with a preset that you will not find errors;
9. The probability of errors in a certain part of the System is proportional to the number of errors already found here;
10. Testing is a challenge to your creative and intellectual abilities. Testing is incredibly creative and intellectual.
This principle, which is of a type obvious when overlooked, is the cause of the most frequent errors in the testing process. Here again psychology intervenes. If you do not understand in advance how the result should look, then something similar to the truth, but not such, will be perceived as a correct result. After all, "we see what we want to see." In other words, despite the destructive nature of testing, somewhere in the depths of the soul, we still keep a desire to see the correct result. But this must be resolutely fought. For example, by encouraging a detailed advance study of all output and expected results. This means that the test case should contain two components:
- input description
- An accurate description of the correct output for the specified input data.
')
Problems can arise when we encounter something that does not have an acceptable description, that looks strange, that does not agree with our expectations or prejudices. When meeting with something similar, we need to have at least some generalized ideas, because if there is not even them, it is easy to miss potential problems altogether. No waiting - no surprises.
Every writer knows - or should know - that editing his own work is a bad idea. They know what a particular chapter of a book should say, for example, and therefore they may not see what it says something else. And they somehow do not want to look for mistakes in their work. The same can be said about the authors of software products.
Another problem is the change of focus. While the programmer is creative in designing and writing code, it can be very difficult to tune into a destructive wave.
Every homeowner knows that shooting wallpapers is very difficult. But if you also hung this wallpaper yourself, it would be unbearably depressing. Similarly with software, most programmers cannot test their programs effectively, since they cannot make mental adjustments that help identify errors. Moreover, the subconscious fear of punishment on the part of colleagues, managers, customers or owners can lead to the avoidance of mistakes.
And another problem: the program may contain errors that are caused by a misunderstanding of the tasks or specifications. And if so, the testing process itself may include derivatives of this misunderstanding.
This does not mean that the way to test your own programs for the programmer is closed. It is simply better (more effective) for someone else to do this. Developers can be helpful participants in the testing team when evaluating the specification and the program code itself.
And further. It was not about the debugging process. In this case, the author of the code is the most effective participant in the process.
The arguments are the same as for the previous principle. The organization that creates software, in many ways as a person: has psychological problems. Moreover, in our best of all worlds, the work of an organization or its managers is often evaluated by the ability to produce programs on time and at a certain cost. The reason for this is that these quantities are easy to quantify, as opposed to reliability, which can be difficult to quantify. Hence, it is difficult for development organizations to be effective in testing their own products. This assessment can be done based on adherence to work schedules and expenses.
It should be noted here that the organization’s participation in the process of testing its own products is not necessarily meaningless. Here we say that it is most successful from an economic point of view to perform such work by an independent party.
This is obvious, but it is often overlooked. We are familiar with a large number of cases where an error could not be detected even when its symptoms were clearly visible in the output lists (results). If expressed differently, errors that are found in subsequent tests are often not noticed in the results of previous tests.
This is a natural and robust tendency to concentrate on acceptable and expected conditions in the testing process. Neglecting unacceptable and unexpected conditions is also natural, but not as sensible. A lot of mistakes can be revealed when a software product is used in some new or unexpected way. It is impossible to determine all the scenarios, how the user will work with the product, but something different from the quiet, familiar flow of the program seems to have every chance of success (in the search for errors).
This is a consequence of the previous principle. The program must also be checked for unwanted third-party effects. For example, software for calculating wages, which issues a salary check correctly, still contains errors, if it also issues additional checks for non-existent employees, or it rewrites the first entry in a personal file.
This problem seems to be often manifested in various systems for testing programs. A typical practice is to sit at the computer and develop tests on the fly, and then carry out these test cases in the program. The bottom line is that test cases are valuable investments that can disappear within a specific environment as soon as the testing process is completed. And whenever there is a need to re-test software (in regression testing or when improving the system), test cases may require new development. This is an additional, difficult job, meeting with which a test engineer can volunteer to avoid it. So it turns out that subsequent efforts to work are rarely as voluminous and of high quality as bulk and qualitative of the initial ones. And if, for example, a change in functionality entails errors, the newly developed tests may not detect them.
A frequent mistake of project managers and a small bell to the fact that someone determines the testing process incorrectly. For example, as a process of demonstrating that a program is working correctly. We accept another definition (although we guess that there are guys who do not do this) - this is the process of executing the program with the persistent intention to find errors. We assert: it is obvious that it is impossible to develop a software product that is completely error free. Even after a well-executed testing and error correction process, it is appropriate to assume that errors still exist; just haven't found them yet.
It may seem that this thing is doubtful, but this phenomenon is observed in many programs. This is how it is on the fingers: suppose that the program consists of modules A and B. In module A 5 errors were found, and in module B only one error was found. And if module A has not yet been thoroughly tested, then the principle says: the probability of finding even more errors in module A is higher than the probability of finding even more errors in module B.
Another way to illustrate this principle is as follows: mistakes have the will to unite into groups, and in a typical case, certain program blocks are more prone to fallacies than others. Although it looks like magic ... nothing supported by magic. It can be useful because it provides some insight and feedback during the testing process. If some part of the program seems to be more error prone than other parts, this phenomenon hints to us that it would be better for us to test here a little longer. And this is a question of investment.
There is even a picture:
Undoubtedly, it is. The creative abilities needed to test a software product exceed (yes it is written there) the creative abilities needed to create this software product. There are many techniques and techniques for the design of heartfelt tests, but all this will not be high, unless creative efforts are made on your part.
1. Mandatory part of testing - determining the expected result;
2. Programmers should avoid testing their own programs (and code sections);
3. Organizations that create programs should avoid testing their own programs;
4. The testing process should include a thorough check of the results of each test;
5. Test cases should be designed for both correct and expected input conditions, as well as for incorrect and unexpected;
6. The study of the System for the fact that it does not do what it should be - only half the story. The second part is to figure out what she is doing wrong;
7. Avoid one-time test cases only if the program itself is not one-time. Disposable test cases for disposable programs. In other cases, those should be avoided;
8. Do not engage in the testing process with a preset that you will not find errors;
9. The probability of errors in a certain part of the System is proportional to the number of errors already found here;
10. Testing is a challenge to your creative and intellectual abilities. Testing is incredibly creative and intellectual.
Principle 1: The mandatory part of testing is determining the expected result.
This principle, which is of a type obvious when overlooked, is the cause of the most frequent errors in the testing process. Here again psychology intervenes. If you do not understand in advance how the result should look, then something similar to the truth, but not such, will be perceived as a correct result. After all, "we see what we want to see." In other words, despite the destructive nature of testing, somewhere in the depths of the soul, we still keep a desire to see the correct result. But this must be resolutely fought. For example, by encouraging a detailed advance study of all output and expected results. This means that the test case should contain two components:
- input description
- An accurate description of the correct output for the specified input data.
')
Problems can arise when we encounter something that does not have an acceptable description, that looks strange, that does not agree with our expectations or prejudices. When meeting with something similar, we need to have at least some generalized ideas, because if there is not even them, it is easy to miss potential problems altogether. No waiting - no surprises.
Principle 2: Programmers should avoid testing their own programs (and code sections).
Every writer knows - or should know - that editing his own work is a bad idea. They know what a particular chapter of a book should say, for example, and therefore they may not see what it says something else. And they somehow do not want to look for mistakes in their work. The same can be said about the authors of software products.
Another problem is the change of focus. While the programmer is creative in designing and writing code, it can be very difficult to tune into a destructive wave.
Every homeowner knows that shooting wallpapers is very difficult. But if you also hung this wallpaper yourself, it would be unbearably depressing. Similarly with software, most programmers cannot test their programs effectively, since they cannot make mental adjustments that help identify errors. Moreover, the subconscious fear of punishment on the part of colleagues, managers, customers or owners can lead to the avoidance of mistakes.
And another problem: the program may contain errors that are caused by a misunderstanding of the tasks or specifications. And if so, the testing process itself may include derivatives of this misunderstanding.
This does not mean that the way to test your own programs for the programmer is closed. It is simply better (more effective) for someone else to do this. Developers can be helpful participants in the testing team when evaluating the specification and the program code itself.
And further. It was not about the debugging process. In this case, the author of the code is the most effective participant in the process.
Principle 3: Programming organizations should avoid testing their own programs.
The arguments are the same as for the previous principle. The organization that creates software, in many ways as a person: has psychological problems. Moreover, in our best of all worlds, the work of an organization or its managers is often evaluated by the ability to produce programs on time and at a certain cost. The reason for this is that these quantities are easy to quantify, as opposed to reliability, which can be difficult to quantify. Hence, it is difficult for development organizations to be effective in testing their own products. This assessment can be done based on adherence to work schedules and expenses.
It should be noted here that the organization’s participation in the process of testing its own products is not necessarily meaningless. Here we say that it is most successful from an economic point of view to perform such work by an independent party.
Principle 4: The testing process should include a thorough check of the results of each test.
This is obvious, but it is often overlooked. We are familiar with a large number of cases where an error could not be detected even when its symptoms were clearly visible in the output lists (results). If expressed differently, errors that are found in subsequent tests are often not noticed in the results of previous tests.
Principle 5: test cases should be designed for both correct and expected input conditions, as well as for incorrect and unexpected
This is a natural and robust tendency to concentrate on acceptable and expected conditions in the testing process. Neglecting unacceptable and unexpected conditions is also natural, but not as sensible. A lot of mistakes can be revealed when a software product is used in some new or unexpected way. It is impossible to determine all the scenarios, how the user will work with the product, but something different from the quiet, familiar flow of the program seems to have every chance of success (in the search for errors).
Principle 6: Researching the System for what it does not do, what it should do, is only half the story. The second part is to figure out what is wrong she does.
This is a consequence of the previous principle. The program must also be checked for unwanted third-party effects. For example, software for calculating wages, which issues a salary check correctly, still contains errors, if it also issues additional checks for non-existent employees, or it rewrites the first entry in a personal file.
Principle 7: one-time test cases for one-time programs, in other cases such tests should be avoided.
This problem seems to be often manifested in various systems for testing programs. A typical practice is to sit at the computer and develop tests on the fly, and then carry out these test cases in the program. The bottom line is that test cases are valuable investments that can disappear within a specific environment as soon as the testing process is completed. And whenever there is a need to re-test software (in regression testing or when improving the system), test cases may require new development. This is an additional, difficult job, meeting with which a test engineer can volunteer to avoid it. So it turns out that subsequent efforts to work are rarely as voluminous and of high quality as bulk and qualitative of the initial ones. And if, for example, a change in functionality entails errors, the newly developed tests may not detect them.
Principle 8: do not engage in the testing process with a preset that you will not find errors
A frequent mistake of project managers and a small bell to the fact that someone determines the testing process incorrectly. For example, as a process of demonstrating that a program is working correctly. We accept another definition (although we guess that there are guys who do not do this) - this is the process of executing the program with the persistent intention to find errors. We assert: it is obvious that it is impossible to develop a software product that is completely error free. Even after a well-executed testing and error correction process, it is appropriate to assume that errors still exist; just haven't found them yet.
Principle 9: the probability of errors in a certain part of the System is proportional to the number of errors already found here.
It may seem that this thing is doubtful, but this phenomenon is observed in many programs. This is how it is on the fingers: suppose that the program consists of modules A and B. In module A 5 errors were found, and in module B only one error was found. And if module A has not yet been thoroughly tested, then the principle says: the probability of finding even more errors in module A is higher than the probability of finding even more errors in module B.
Another way to illustrate this principle is as follows: mistakes have the will to unite into groups, and in a typical case, certain program blocks are more prone to fallacies than others. Although it looks like magic ... nothing supported by magic. It can be useful because it provides some insight and feedback during the testing process. If some part of the program seems to be more error prone than other parts, this phenomenon hints to us that it would be better for us to test here a little longer. And this is a question of investment.
There is even a picture:
Principle 10: Testing is a challenge to your creative and intellectual abilities. Testing is incredibly creative and intelligent
Undoubtedly, it is. The creative abilities needed to test a software product exceed (yes it is written there) the creative abilities needed to create this software product. There are many techniques and techniques for the design of heartfelt tests, but all this will not be high, unless creative efforts are made on your part.
Source: https://habr.com/ru/post/330746/
All Articles