Using the rejected defect rate to improve the error report
Good Friday everyone, friends! At the end of June, we are launching a new group on the QA-Specialist course, and this will be the subject of today's publication.
There are many indicators by which you can measure the effectiveness of the team of testers. One of them is the Rejected Defect Ratio ratio, or the number of rejected error reports divided by the total number of reports received. You must be thinking that if the number of rejected reports is zero, then this is good, but not everything is so simple. Let's look at the types of rejected errors, see how they affect the ratio of rejected errors, and calculate the correct ratio for your team.
')
There are three categories of rejected errors:
Let's start with the mistakes themselves.
There are two types of non-reproducible errors. The first is a mistake that is really difficult to reproduce. This may be an error resulting from the interaction of several parameters, some of which you do not even know.
Suppose you run several tests in a row, and one of the tests changed the configuration parameter from the default value A to some other value B. The error occurs only when the configuration parameter contains the value B, and the input value is C. At If you attempt to reproduce the error, you will most likely want to start with a known state in order to initialize the system (or, possibly, perform a clean installation). No error will occur, as the configuration parameter now now contains the default value of A.
Another variant of this kind of non-reproducible error is when the test actually detected a defect, but there is no data in the playback information: a step, a specific input value, or an understanding that an error occurs only in a specific course of action. As a result, attempts to reproduce the error lead to nothing.
In both of the above cases, however, there is indeed a defect in the product itself.
The second type of non-reproducible error is when the error cannot be repeated, because there is none. The tester might have noticed something, but interpreted it incorrectly, or there could be some problem in the system used for testing, such as a faulty hardware component, an incompatible driver, or incorrect access settings. Attempts to reproduce the error in a properly configured system fail.
Both of these types of errors are usually marked in the error reporting systems as “rejected - cannot be reproduced.”
This type of error occurs if the tester decides that the product should behave in a certain way and reports an error when the behavior did not meet its expectations. However, a more detailed study of the requirements shows that the tester's expectations were erroneous, and the product actually functioned correctly. That is, the tested product worked correctly, and the tester, who was not sufficiently familiar with the requirements, was mistaken.
Such errors are usually marked in systems of error reports as “rejected - not error” or “rejected — by architecture” (that is, the behavior corresponds to the architecture).
Repetitive errors are those that someone has already reported, and the next one reports. An error is repeated only if the "symptoms" of its appearance are the same. And if the root cause of the error is the same, but the “symptoms” were different, it is not a repetition of the error!
These errors are usually marked in the error reporting systems as “Rejected - Duplicate / Repeat.”
Obviously, an incorrect error is a kind of waste of time that the tester spent on reproducing the error and reporting it, the time that those who sort the errors spend reading and understanding them, and the time the developers spend trying to reproduce an unrepeatable error or for the correction (and non-correction) of something that did not need this correction.
In addition to the fact that the rejected error rate or RDR is a measure of the inefficiency of the testers team, it also speaks of the professionalism of testers in general. An error that cannot be reproduced due to the lack of necessary information in the report indicates that the testers were not meticulous and did not work hard enough to reproduce this error using the steps described by them. In addition, for errors that are reproduced infrequently, testers generally did not make a note about the low frequency of reproduction in the report.
The appearance of an incorrect error indicates that testers do not fully understand the requirements for the product. Repeated errors indicate that testers did not perform a minimum search in the local database of errors to check whether it occurred before. Or, it means that the specialist who reported the error first did not include the right keywords in the title to facilitate the search for other colleagues.
In turn, if it turns out that the error I found is rejected, I am indignant, because I was considered a non-professional. On the one hand, this means that I will defend the errors found. When my report is rejected, I do the following:
On the other hand, a certain probability of error rejection makes me be careful. If I’m not completely sure I’ve found a bug, I’ll spend some more time checking before reporting. I often ask a colleague whether I interpret the requirements correctly, or check whether the error is reproduced on a foreign system.
The testing team should monitor and strive to reduce the level of RDR. The only question is what is considered good RDR?
At first glance, it seems that 0% is the optimal result, but I strongly disagree with this. I believe that when the RDR is kept at some healthy level is normal, because if it is close to zero, the team of testers obviously suffers from no less serious problems than, say, too high RDR.
The testing team must make great efforts to achieve extremely low RDR. Each rejected error will be analyzed to understand what went wrong, and each tester who reported an error that was rejected will have to clarify what really happened and how this situation can be avoided in the future. As a result, testers will report errors in which they are absolutely sure.
If they notice behavior that they believe will damage the usability of the product, they will prefer to take such behavior for granted, rather than justifying themselves that they have found a mistake, which in fact, based on the requirements, is not an error. If they have evidence that an error has occurred, but there is no good scenario for its reproduction, they will prefer not to report it; they really don't want to upset themselves. If they encounter a non-serious bug, they may decide not to report it at all, because minor bugs do not always fix them, so why risk and fear that the error you found will be rejected?
In short, striving for a very low RDR gives rise to stress and unhealthy behavior in the testers team, and also increases the likelihood that some errors will go unnoticed.
We need testers who not only report obvious errors, but also warn of any suspicious behavior in the project. We believe that testers who attach great importance to avoiding errors, even at the cost of duplicates in reports, are better than testers who spend hours checking whether a bug has already been fixed in reports earlier or not, in fear that they make a duplicate. We want testers to feel comfortable questioning a system architect’s word or specification of requirements, even if this means that some of their mistakes will be flagged as rejected.
We need testers who are not afraid to make mistakes from time to time. This means that you need a balance, so some small RDR is considered acceptable.
My rule of thumb is that RDR should be 15 percent. This value is based on my experience with the testers team, which, by all accounts, was a good and effective team. This was our RDR during several projects that followed one after another, while the other team that worked on the same projects and in parallel with us, although it was less aware of the product and was considered less effective, had a 30 — percent RDR .
I do not think that there is any justification for this value, except for my inner feeling. This is definitely not scientific. I will not argue with a team that aims at 10 or 20 percent, but I think that putting up with 30 percent or setting a goal of 5 percent is already a problem.
In the end, this is a decision that must be made by a team of testers, based on product features, team level of expertise, development model, development team experience and much more. I highly recommend that you monitor RDR and think about whether you need to do something with it. And if it is too high or low, appropriate measures should be taken.
By established tradition, we are waiting for your comments and invite you to a free webinar , which will be held on June 14th. See you!
There are many indicators by which you can measure the effectiveness of the team of testers. One of them is the Rejected Defect Ratio ratio, or the number of rejected error reports divided by the total number of reports received. You must be thinking that if the number of rejected reports is zero, then this is good, but not everything is so simple. Let's look at the types of rejected errors, see how they affect the ratio of rejected errors, and calculate the correct ratio for your team.
')
There are three categories of rejected errors:
- Unrecoverable errors;
- Incorrect errors;
- Repetitive errors.
Let's start with the mistakes themselves.
Unrecoverable errors
There are two types of non-reproducible errors. The first is a mistake that is really difficult to reproduce. This may be an error resulting from the interaction of several parameters, some of which you do not even know.
Suppose you run several tests in a row, and one of the tests changed the configuration parameter from the default value A to some other value B. The error occurs only when the configuration parameter contains the value B, and the input value is C. At If you attempt to reproduce the error, you will most likely want to start with a known state in order to initialize the system (or, possibly, perform a clean installation). No error will occur, as the configuration parameter now now contains the default value of A.
Another variant of this kind of non-reproducible error is when the test actually detected a defect, but there is no data in the playback information: a step, a specific input value, or an understanding that an error occurs only in a specific course of action. As a result, attempts to reproduce the error lead to nothing.
In both of the above cases, however, there is indeed a defect in the product itself.
The second type of non-reproducible error is when the error cannot be repeated, because there is none. The tester might have noticed something, but interpreted it incorrectly, or there could be some problem in the system used for testing, such as a faulty hardware component, an incompatible driver, or incorrect access settings. Attempts to reproduce the error in a properly configured system fail.
Both of these types of errors are usually marked in the error reporting systems as “rejected - cannot be reproduced.”
Incorrect errors
This type of error occurs if the tester decides that the product should behave in a certain way and reports an error when the behavior did not meet its expectations. However, a more detailed study of the requirements shows that the tester's expectations were erroneous, and the product actually functioned correctly. That is, the tested product worked correctly, and the tester, who was not sufficiently familiar with the requirements, was mistaken.
Such errors are usually marked in systems of error reports as “rejected - not error” or “rejected — by architecture” (that is, the behavior corresponds to the architecture).
Repeated errors
Repetitive errors are those that someone has already reported, and the next one reports. An error is repeated only if the "symptoms" of its appearance are the same. And if the root cause of the error is the same, but the “symptoms” were different, it is not a repetition of the error!
These errors are usually marked in the error reporting systems as “Rejected - Duplicate / Repeat.”
How rejected errors affect a command
Obviously, an incorrect error is a kind of waste of time that the tester spent on reproducing the error and reporting it, the time that those who sort the errors spend reading and understanding them, and the time the developers spend trying to reproduce an unrepeatable error or for the correction (and non-correction) of something that did not need this correction.
In addition to the fact that the rejected error rate or RDR is a measure of the inefficiency of the testers team, it also speaks of the professionalism of testers in general. An error that cannot be reproduced due to the lack of necessary information in the report indicates that the testers were not meticulous and did not work hard enough to reproduce this error using the steps described by them. In addition, for errors that are reproduced infrequently, testers generally did not make a note about the low frequency of reproduction in the report.
The appearance of an incorrect error indicates that testers do not fully understand the requirements for the product. Repeated errors indicate that testers did not perform a minimum search in the local database of errors to check whether it occurred before. Or, it means that the specialist who reported the error first did not include the right keywords in the title to facilitate the search for other colleagues.
In turn, if it turns out that the error I found is rejected, I am indignant, because I was considered a non-professional. On the one hand, this means that I will defend the errors found. When my report is rejected, I do the following:
- I check again if the error is reproduced in my system and add playback steps if I missed something;
- If my lack of understanding of the requirements was caused by an ambiguous requirement or incorrect documentation, I will insist that the error be marked as a documentation error and closed only when the documentation is corrected;
- If I believe that the product’s behavior when fulfilling a requirement is wrong, I will talk about requirements with architects and developers, try to convince them that the requirements need to be updated (in the end, I represent the customer’s opinion!);
- If the error is rejected as a duplicate, I will make sure that it was not marked the same way, or that it does not appear "in the same scenario."
On the other hand, a certain probability of error rejection makes me be careful. If I’m not completely sure I’ve found a bug, I’ll spend some more time checking before reporting. I often ask a colleague whether I interpret the requirements correctly, or check whether the error is reproduced on a foreign system.
Opinion against the total absence of rejected errors
The testing team should monitor and strive to reduce the level of RDR. The only question is what is considered good RDR?
At first glance, it seems that 0% is the optimal result, but I strongly disagree with this. I believe that when the RDR is kept at some healthy level is normal, because if it is close to zero, the team of testers obviously suffers from no less serious problems than, say, too high RDR.
The testing team must make great efforts to achieve extremely low RDR. Each rejected error will be analyzed to understand what went wrong, and each tester who reported an error that was rejected will have to clarify what really happened and how this situation can be avoided in the future. As a result, testers will report errors in which they are absolutely sure.
If they notice behavior that they believe will damage the usability of the product, they will prefer to take such behavior for granted, rather than justifying themselves that they have found a mistake, which in fact, based on the requirements, is not an error. If they have evidence that an error has occurred, but there is no good scenario for its reproduction, they will prefer not to report it; they really don't want to upset themselves. If they encounter a non-serious bug, they may decide not to report it at all, because minor bugs do not always fix them, so why risk and fear that the error you found will be rejected?
In short, striving for a very low RDR gives rise to stress and unhealthy behavior in the testers team, and also increases the likelihood that some errors will go unnoticed.
We need testers who not only report obvious errors, but also warn of any suspicious behavior in the project. We believe that testers who attach great importance to avoiding errors, even at the cost of duplicates in reports, are better than testers who spend hours checking whether a bug has already been fixed in reports earlier or not, in fear that they make a duplicate. We want testers to feel comfortable questioning a system architect’s word or specification of requirements, even if this means that some of their mistakes will be flagged as rejected.
We need testers who are not afraid to make mistakes from time to time. This means that you need a balance, so some small RDR is considered acceptable.
Finding the optimal ratio of rejected defects
My rule of thumb is that RDR should be 15 percent. This value is based on my experience with the testers team, which, by all accounts, was a good and effective team. This was our RDR during several projects that followed one after another, while the other team that worked on the same projects and in parallel with us, although it was less aware of the product and was considered less effective, had a 30 — percent RDR .
I do not think that there is any justification for this value, except for my inner feeling. This is definitely not scientific. I will not argue with a team that aims at 10 or 20 percent, but I think that putting up with 30 percent or setting a goal of 5 percent is already a problem.
In the end, this is a decision that must be made by a team of testers, based on product features, team level of expertise, development model, development team experience and much more. I highly recommend that you monitor RDR and think about whether you need to do something with it. And if it is too high or low, appropriate measures should be taken.
By established tradition, we are waiting for your comments and invite you to a free webinar , which will be held on June 14th. See you!
Source: https://habr.com/ru/post/455240/
All Articles