Report from SQA Days - Test design: easier to read or easier to write
Alexander Alexandrov , the “grandfather of testing” in the CIS, made a report at the 15th anniversary SQA Days in Moscow.
Slides
www.slideshare.net/VLDCORP/ss-33747358
Video of performance:
')
Performance plan
1.Test Design
2. How to check testability requirements
3. Requirements and testing without test cases and / or without testers
4. Requirements and changes
5. Easier to read or easier to write.
6. How are test cases arranged?
7.What difficulties arise
8. How to overcome them
To begin with, there is an increase in the general interest in automation, this is a trend and that is good! But the need for a “foundation” for the development of automated test scripts is increasing. After all, if there is no such foundation, then we get the complexity of creating, using and maintaining scripts. It is not clear what they check, it is not clear how they check and it is not clear how they are related to the requirements! This situation is well known to those who develop these scripts and to those who use these scripts, as well as to those who believe the information that provides these scripts.
Also the next important thesis is the importance of all of the above, outside the context of test automation . Replace all that concerns automatic testing for manual testing, and you get exactly the same thing!
So what do we want?
We want to simplify and speed up work, as well as increase the reliability of work results for:
- Tester;
- Developer of automated test scripts;
- Test manager analyzing test results (manual / automated) / customer / future users, etc.
My report will be of two unequal parts. In the first one, I remind you what a test design is, and in the second, how to get around errors and less often step on the same “rake”.
How to check testability requirements
Requirements mantras:
Every requirement must be verified. There are basic criteria to be guided. A person may make mistakes when checking requirements. Therefore, you need to know how to reliably check them! Please note, I do not say perfect check! It's impossible! At least reliably check!
1 method - Early design of test cases
That is, as soon as you start using UML diagrams, a story that was scanned several times, with remarks, the design begins, then you should ask questions: what is the limit on this field, and what will be after this action, and who should do it, and what is the default period. Due to this, it is quite easy to detect many gaps at the design stage of test cases.
2 method - Visualization of the connection of test cases and requirements (where and how this feature is checked)
Because it may turn out that you wrote 3000 test cases, everything is absolutely wonderful, but we just didn’t check some important things. The reasons are different: they forgot, they did not understand, they thought that they had done the “obvious” way for us, they missed something. I point this out, not because I have a one-stop solution, but because we do it every day!
The next aspect, why do we need anything besides requirements?
Let's compare! The difference at first glance is very simple, but in fact, it is deep! The idea of why testing requirements.
Requirements vs. Testing
Compliance with the software product requirements should answer the following questions:
? Are all requirements implemented?
? Are all requirements implemented correctly?
? Is there too much?
? Is the diagnosis adequate?
“Why do you write 1000 cases, and would not it be easier to take a tester, give him the requirements to test for them?” I, as a test manager, come across this question regularly. And I have to answer it all the time in different ways. Just recently, I answered the same question: “Here I have a serious complex project for a year, only 2 testers participate in it, who tested the requirements themselves and wrote cases. During the year, the testers left and the new ones, thanks to their test cases and the analysis done, were able to take over the essence of the project and continue working on it. ”
Often, I hear that they offer to conduct testing, but without test cases and / or without testers.
Requirements and testing without test cases and / or without testers
- Testing by developers - problems are well known (subject awareness, ignorance of the whole project, logic)
- Testing by analysts - do not need test cases? (an interesting idea, but always testing a positive script). Only what works as it should in the requirements is checked, even small deviations from the correct behavior are not considered. A linear case is checked, but a bundle of requirements is not performed for integration testing)
Of course, in such cases, the quality and volume of testing vary greatly, testing takes only 20% -30% of the total possible volume + serious errors in the requirements are omitted.
Another important aspect that I would like to draw attention to is that we always say and imply a situation where we are in a development project and we have a stable team. There is a team that is professional and works together. But it so happens that you are given a project to support. Completely different team in another country. You do not know these people at all, and, of course, there are problems with requirements. As there is a lot of development work, little is done about the system requirements, and the maintenance becomes more. All this should take into account a number of features.
Requirements and changes
The aspect of test design is also important here, as if every time in the project that you are given to accompany you, you will make a riddle: “How can I test a change?”, Then you will spend a lot of time and some attempts may be completely unsuccessful . At the same time, the work on making a test design done once before changing it may well help you.
And the penultimate aspect of the general part, which we will now finish, is data-driven testing :
And:
There are separate test case steps and there are separate data that are submitted to the test case. The data can vary, the steps of the test case are stable, and we can thereby check for a large number of situations. Not to mention that testing situation, data-driven is very important in cases where you have a data-dependent effect.
And one more aspect that I would like to mention is the granularity of the requirements . Details may be more or less detailed. But what nuances arise, if the requirements are detailed, it is difficult to maintain, but just to use. And if there are few details, it is difficult to use, and it is easier to maintain. You can find a reasonable compromise: checklists and testing for the requirements , but it will always contain risks that are well known.
Let us turn to the second part of my report.
The structure of the test case is well known, it is described below and nothing new humanity came up with
What is written about the structure of the test case
Test case includes:
- Format;
- Content;
The format is the same everywhere;
- The sequence number of the step;
- Impact on the system;
- Expected Result;
And remember! The devil is in the details!
If we make a comparison of formats, then we will see that all of them have the same meaningful content. Analysis and comparison of formats:
Here you can see one example.
Pay attention to the table, we have the sequence number, the action, the data that we enter and the expected result. And what do we see?
First, we see the expected result is not there, then we see a not entirely correct description of the action (“if the button is unavailable, then click” - the question is whether the button should be available or not?), The numbers in the expected result are incorrectly indicated (if the three-digit number, fractional part, etc.
Formally, all aspects are included, but we cannot get reliable information about the quality of the product due to these actions, since the necessary tests may be missed.
It is well known what data-driven testing is . But we offer it in more detail. Data expansion is easy.
Separation of steps from data
1. Separate description with links
2. Link data and expected results
Steps
1. Actions to perform a test case
2. Navigation rules
Data
1. What the user enters and / or selects and / or clicks (field, list, ...)
2. Expected Results
The structure of the test case (clarification) . Change the format:
And what do we get as a result of such changes?
Everything here is absolutely wonderful, except that it says “enter data”, besides the frequent use of terms such as “any”, “correct”, “expected”, etc.
Our next suggestion is on the test case structure . Getting rid of:
Notice, is there anything else left?
Yes, this is a link to an external document, which the tester will have to contact to obtain confirmation of the correct expected result. Here we recall the name of the report "Test Design: Easier to Read or Easier to Write"
What do I suggest?
I would like to pay special attention to data preparation. It is clear that the test case can be performed in any situation. The system must be in a certain state and this state must be filled. There are cases when, in order to prepare the data, you need to perform some other actions, including several hundred steps. Therefore, it is very important to register the data with which you work. And even better, if you have the opportunity to get exactly the data with which the customer will work.
Data :
1. Roles, meanings
2. Links to the data warehouse (requests)
Data preparation :
1. Preconditions (database state)
2. Execute SQL queries
3. Actions in the format of test cases
The proposed solutions are a reasonable compromise of the complexity of steps and data sets. Sometimes it’s worth writing two or three similar test cases, reducing the volume of data sets by an order of magnitude (due to repetitions). All this will allow the tester not to stray and miss something, and it will be easier for the developer to build the display of the script code on the test case description (necessary when analyzing and maintaining the scripts).
Yes, the test design, we complicate their work, but we increase its quality. Test design is the most difficult thing to test. There are no two similar test designs. In most cases, in my practice, the introduction of a test design on an ongoing basis saves the labor costs of testing a manual and automatic tester. And, of course, it requires a highly skilled test designer. You can take the tester and even the developer and train, then you can get a good test designer. Just from the street man does not teach this.
Slides
www.slideshare.net/VLDCORP/ss-33747358
Video of performance:
')
Performance plan
1.Test Design
2. How to check testability requirements
3. Requirements and testing without test cases and / or without testers
4. Requirements and changes
5. Easier to read or easier to write.
6. How are test cases arranged?
7.What difficulties arise
8. How to overcome them
To begin with, there is an increase in the general interest in automation, this is a trend and that is good! But the need for a “foundation” for the development of automated test scripts is increasing. After all, if there is no such foundation, then we get the complexity of creating, using and maintaining scripts. It is not clear what they check, it is not clear how they check and it is not clear how they are related to the requirements! This situation is well known to those who develop these scripts and to those who use these scripts, as well as to those who believe the information that provides these scripts.
Also the next important thesis is the importance of all of the above, outside the context of test automation . Replace all that concerns automatic testing for manual testing, and you get exactly the same thing!
So what do we want?
We want to simplify and speed up work, as well as increase the reliability of work results for:
- Tester;
- Developer of automated test scripts;
- Test manager analyzing test results (manual / automated) / customer / future users, etc.
My report will be of two unequal parts. In the first one, I remind you what a test design is, and in the second, how to get around errors and less often step on the same “rake”.
How to check testability requirements
Requirements mantras:
- Completeness
- Consistency
- Unambiguity
- Traceability
- Feasibility
- Testability
- ...
Every requirement must be verified. There are basic criteria to be guided. A person may make mistakes when checking requirements. Therefore, you need to know how to reliably check them! Please note, I do not say perfect check! It's impossible! At least reliably check!
1 method - Early design of test cases
That is, as soon as you start using UML diagrams, a story that was scanned several times, with remarks, the design begins, then you should ask questions: what is the limit on this field, and what will be after this action, and who should do it, and what is the default period. Due to this, it is quite easy to detect many gaps at the design stage of test cases.
2 method - Visualization of the connection of test cases and requirements (where and how this feature is checked)
Because it may turn out that you wrote 3000 test cases, everything is absolutely wonderful, but we just didn’t check some important things. The reasons are different: they forgot, they did not understand, they thought that they had done the “obvious” way for us, they missed something. I point this out, not because I have a one-stop solution, but because we do it every day!
The next aspect, why do we need anything besides requirements?
Let's compare! The difference at first glance is very simple, but in fact, it is deep! The idea of why testing requirements.
Requirements vs. Testing
- Requirements : determine what should work and how
- Testing : Determine what does not work or does not work the way it should work
Compliance with the software product requirements should answer the following questions:
? Are all requirements implemented?
? Are all requirements implemented correctly?
? Is there too much?
? Is the diagnosis adequate?
“Why do you write 1000 cases, and would not it be easier to take a tester, give him the requirements to test for them?” I, as a test manager, come across this question regularly. And I have to answer it all the time in different ways. Just recently, I answered the same question: “Here I have a serious complex project for a year, only 2 testers participate in it, who tested the requirements themselves and wrote cases. During the year, the testers left and the new ones, thanks to their test cases and the analysis done, were able to take over the essence of the project and continue working on it. ”
Often, I hear that they offer to conduct testing, but without test cases and / or without testers.
Requirements and testing without test cases and / or without testers
- Testing by developers - problems are well known (subject awareness, ignorance of the whole project, logic)
- Testing by analysts - do not need test cases? (an interesting idea, but always testing a positive script). Only what works as it should in the requirements is checked, even small deviations from the correct behavior are not considered. A linear case is checked, but a bundle of requirements is not performed for integration testing)
Of course, in such cases, the quality and volume of testing vary greatly, testing takes only 20% -30% of the total possible volume + serious errors in the requirements are omitted.
Another important aspect that I would like to draw attention to is that we always say and imply a situation where we are in a development project and we have a stable team. There is a team that is professional and works together. But it so happens that you are given a project to support. Completely different team in another country. You do not know these people at all, and, of course, there are problems with requirements. As there is a lot of development work, little is done about the system requirements, and the maintenance becomes more. All this should take into account a number of features.
Requirements and changes
- Obligatory change
- "Surprise" of change
- Ways to commit changes
- Matrix of communication requirements and test cases (assessment of labor costs for the implementation and testing of changes)
- Impact of changes on the application as a whole
The aspect of test design is also important here, as if every time in the project that you are given to accompany you, you will make a riddle: “How can I test a change?”, Then you will spend a lot of time and some attempts may be completely unsuccessful . At the same time, the work on making a test design done once before changing it may well help you.
And the penultimate aspect of the general part, which we will now finish, is data-driven testing :
- The expected results are displayed in response to correctly entered data.
- Adequate response to incorrect data , including those not stated in the requirements (corresponding error messages)
And:
- Equivalence classes
- Boundary values
There are separate test case steps and there are separate data that are submitted to the test case. The data can vary, the steps of the test case are stable, and we can thereby check for a large number of situations. Not to mention that testing situation, data-driven is very important in cases where you have a data-dependent effect.
And one more aspect that I would like to mention is the granularity of the requirements . Details may be more or less detailed. But what nuances arise, if the requirements are detailed, it is difficult to maintain, but just to use. And if there are few details, it is difficult to use, and it is easier to maintain. You can find a reasonable compromise: checklists and testing for the requirements , but it will always contain risks that are well known.
Let us turn to the second part of my report.
The structure of the test case is well known, it is described below and nothing new humanity came up with
What is written about the structure of the test case
Test case includes:
- Format;
- Content;
The format is the same everywhere;
- The sequence number of the step;
- Impact on the system;
- Expected Result;
And remember! The devil is in the details!
If we make a comparison of formats, then we will see that all of them have the same meaningful content. Analysis and comparison of formats:
- Essentially the same:
- You can still search the books, the Internet ...
- Is the content adequate to the format?
Here you can see one example.
Pay attention to the table, we have the sequence number, the action, the data that we enter and the expected result. And what do we see?
First, we see the expected result is not there, then we see a not entirely correct description of the action (“if the button is unavailable, then click” - the question is whether the button should be available or not?), The numbers in the expected result are incorrectly indicated (if the three-digit number, fractional part, etc.
Formally, all aspects are included, but we cannot get reliable information about the quality of the product due to these actions, since the necessary tests may be missed.
It is well known what data-driven testing is . But we offer it in more detail. Data expansion is easy.
Separation of steps from data
1. Separate description with links
2. Link data and expected results
Steps
1. Actions to perform a test case
2. Navigation rules
Data
1. What the user enters and / or selects and / or clicks (field, list, ...)
2. Expected Results
The structure of the test case (clarification) . Change the format:
- Step number
- Impact on the system
- Data link (what is the link you need)
- Expected result (possibly a link to data and / or illustration)
And what do we get as a result of such changes?
Everything here is absolutely wonderful, except that it says “enter data”, besides the frequent use of terms such as “any”, “correct”, “expected”, etc.
Our next suggestion is on the test case structure . Getting rid of:
- Loop type “Repeat steps 5-73 for all possible data”
- Designs like "any", "appropriate". "Suitable", "expected" without the necessary clarifications, for example:
Notice, is there anything else left?
Yes, this is a link to an external document, which the tester will have to contact to obtain confirmation of the correct expected result. Here we recall the name of the report "Test Design: Easier to Read or Easier to Write"
What do I suggest?
I would like to pay special attention to data preparation. It is clear that the test case can be performed in any situation. The system must be in a certain state and this state must be filled. There are cases when, in order to prepare the data, you need to perform some other actions, including several hundred steps. Therefore, it is very important to register the data with which you work. And even better, if you have the opportunity to get exactly the data with which the customer will work.
Data :
1. Roles, meanings
2. Links to the data warehouse (requests)
Data preparation :
1. Preconditions (database state)
2. Execute SQL queries
3. Actions in the format of test cases
The proposed solutions are a reasonable compromise of the complexity of steps and data sets. Sometimes it’s worth writing two or three similar test cases, reducing the volume of data sets by an order of magnitude (due to repetitions). All this will allow the tester not to stray and miss something, and it will be easier for the developer to build the display of the script code on the test case description (necessary when analyzing and maintaining the scripts).
Yes, the test design, we complicate their work, but we increase its quality. Test design is the most difficult thing to test. There are no two similar test designs. In most cases, in my practice, the introduction of a test design on an ongoing basis saves the labor costs of testing a manual and automatic tester. And, of course, it requires a highly skilled test designer. You can take the tester and even the developer and train, then you can get a good test designer. Just from the street man does not teach this.
Source: https://habr.com/ru/post/242385/
All Articles