Experience with TDD and thinking about how to test code
Our company recently hosted a lecture “Engineering Practices” and this turned out to be an introduction to TDD.
Oh no not this! "Sometimes they come back" (c) Stephen King
In previous work, we spent 3 years trying to introduce this technique. It was painful. Management sincerely believed that TDD would solve the problems of the company. Reality is strikingly inconsistent with this. All this awakened the erased memories of the Soviet era. I remembered posters “Forward to the victory of communism” hanging on the walls and phrases like “The doctrine of Marx is omnipotent because it is true”.
So what's wrong with the TDDConservatory ?
')
Unfortunately, my post was understood by many in the sense that I am against testing. And especially against unit tests in all their manifestations. This is not quite true; more precisely, this is not at all true.
I copy here an important remark, which was at the end of the post and it was not too striking.
There are 3 types of code when unit tests are very appropriate and their use is justified:
If you easily and simply write unit tests, which means that your code is easy to isolate, then you don’t have to read further and understand my experience with pain, sweat and blood. Well, if not, then keep in mind that the following is an exceptional single case, but it is well-worn and meaningful.
For a start, we will define the terms.
Manual tests
The functionality is tested manually on a “live” system using the standard UI for it.
This is the usual, traditional QA testing method.
Functional tests
The programmer writes a kind of "test" UI, which allows you to run certain scripts on a "live" system, with the interaction of these modules.
This is a very common testing method used by programmers.
Unit tests
The programmer writes tests that are able to run in an “isolated” environment - without other modules. Other modules to which the test code should refer to during the work are replaced by mocks. Thus, tests can be executed automatically, on the build machine, after compilation.
There is a trend for the use of such tests. And programmers are under pressure (moral and administrative) in order to switch to their use.
Separate remark about TDD
TDD is a programming technique that uses unit tests extensively. And if something is wrong with the Unit tests, then TDD can add to this trouble (which it successfully does), but it does not diminish.
"Error in the code"
The code is written with an error and does something different from the programmer’s intent.
"Integration errors"
Two programmers wrote interacting modules / classes, and misunderstood the protocol. A protocol means both parameters passed to functions and a valid / valid sequence of calls.
Moreover, each of the classes can be written “purely”, but when combined, they give a teravilny system operation.
"Multisulting and Timing Errors"
If several threads participate in the work, there may be errors associated with access to a shared resource or with a sequence of calls. These are the most elusive (after memory corruption) bugs that I have encountered.
“UI Errors” and “Unexpected Errors”
The UI does not meet the requirements at all or somehow crookedly handles some specific situations.
For example, it allows for the input of incorrect data or does not display a sufficiently clear message in case of a negative result.
According to my experience (I don’t have accurate statistics, unfortunately), “errors in the code” make up less than 30% of all bugs in a fairly complex system. If the programmer is experienced enough, the number of such errors will be even less.
The vast majority of such errors are perfectly captured using “code reviews” (code review) and functional tests.
Unit tests are capable of testing code with a wide variety of possible options, up to all conceivable input data. You can test the code very carefully, you can even "to the last comma."
The problem is that there is a certain set of situations that can happen in a real system. Testing something beyond this set is irrelevant and is a waste of time.
Typically, functional tests cover a completely rational set of possible situations.
The strongest side of unit tests is automation.
But personally, I (over 3 years of work) did not come across a situation where unit tests helped find a bug quite a long time after it was written.
That is, unit tests are effective (helping to find bugs) for an exceptionally short time after writing.
I have never seen any unit tests run automatically on the builder unit. They are there "clean clean." And, of course, they serve the guidance of peace of mind. :)
Minus unit tests are a huge effort to spend on writing and programming mocks. That is why the time required to develop using unit tests is 2-5 times longer than without them.
Of course, if the code weakly uses other modules, so that it is very easy to isolate it completely, then the unit tests almost intersect with the functional tests and the difference between them is only in the name and minor technical details.
Summary:
Unit tests do not give a significant gain in system reliability.
There are 3 types of code when unit tests are very appropriate and their use is justified:
If the cost of the error is exceptionally high (for example, we write the software of an airplane or a power plant), then, of course, no costs are excessive and unit tests must be used. But usually the cost of the error is much less.
The backfill question is: who and how can make sure that the programmer wrote unit tests well enough?
There are two possible options for testing unit tests:
Instrumental check
It is possible to measure the "coverage" of the code with tests - whether all the lines of code were executed as part of tests. I have never seen such a check used. Well, of course, such tests can verify that the tests "cover" the code, but they cannot verify that the tests "cover" the possible input data.
Inspection of unit tests by another programmer or supervisor
I have seen it done. Writing unit tests is a very boring thing. Understanding them is generally dark. That is, you can understand, but it requires a serious amount of time and motivation. At the very beginning of implementation, people had an interest and motivation to do everything as it should to figure out what it was - unit tests and TDD. But it passed very soon.
Thus, it is quite difficult to control the programmer that he wrote unit tests well.
As soon as control and administrative pressure decreases, programmers, here are lazy people, tend to write unit tests more and more “for show” and test their code using manual or functional testing.
For systems with a not too high error cost, programmers do not have enough pressure to be paranoid for honest use of unit tests.
If the programmer misses something in the code, it is caught in QA or already at the client and the bug is returned for correction. Feedback makes you take the code seriously and attentively.
If a programmer misses something in writing tests, in fact, no one will ever notice. The lack of feedback allows you to treat the question of disregard.
Even if the programmer got five times more time to develop,I ’d rather go to read Habrahabr instead of jigging tests, moki, tests, moki ... and so many days in a row.
How else can you make unit tests "work"? They should work if the tests are written by another programmer, and it’s best to the code itself.
And here we come to TDD.
If the test is written before and provided to the programmer as source data, it becomes a “technical requirement” (requirement). Of course, the one who writes the test must write and program all the required mocks in advance. Haha, I would not want to be the programmer whose responsibility it is to program tests for others.
Well, can a programmer himself write tests to his code in advance?
In my experience, no and no. Programmers (and I and the vast majority with whom I discussed it) just think in the opposite direction.
Yes, at first people tried to honestly follow the methodology. But after some time, they started to write the code first and then add tests to it. And after some time, the tests lost their versatility and meticulousness. And then “shtob was” was written at all, since they ask.
Does it even need it?
Oh no not this! "Sometimes they come back" (c) Stephen King
In previous work, we spent 3 years trying to introduce this technique. It was painful. Management sincerely believed that TDD would solve the problems of the company. Reality is strikingly inconsistent with this. All this awakened the erased memories of the Soviet era. I remembered posters “Forward to the victory of communism” hanging on the walls and phrases like “The doctrine of Marx is omnipotent because it is true”.
So what's wrong with the TDD
')
Disclaimer
Unfortunately, my post was understood by many in the sense that I am against testing. And especially against unit tests in all their manifestations. This is not quite true; more precisely, this is not at all true.
I copy here an important remark, which was at the end of the post and it was not too striking.
There are 3 types of code when unit tests are very appropriate and their use is justified:
- Software of increased reliability - for example, software for aircraft, power plants, etc.
- Easily isolated code - algorithms and all that.
- Utilities is a code that will be VERY widely used in the system, so it’s worth thoroughly scanning them in advance
If you easily and simply write unit tests, which means that your code is easy to isolate, then you don’t have to read further and understand my experience with pain, sweat and blood. Well, if not, then keep in mind that the following is an exceptional single case, but it is well-worn and meaningful.
For a start, we will define the terms.
Terminology
Manual tests
The functionality is tested manually on a “live” system using the standard UI for it.
This is the usual, traditional QA testing method.
Functional tests
The programmer writes a kind of "test" UI, which allows you to run certain scripts on a "live" system, with the interaction of these modules.
This is a very common testing method used by programmers.
Unit tests
The programmer writes tests that are able to run in an “isolated” environment - without other modules. Other modules to which the test code should refer to during the work are replaced by mocks. Thus, tests can be executed automatically, on the build machine, after compilation.
There is a trend for the use of such tests. And programmers are under pressure (moral and administrative) in order to switch to their use.
Separate remark about TDD
TDD is a programming technique that uses unit tests extensively. And if something is wrong with the Unit tests, then TDD can add to this trouble (which it successfully does), but it does not diminish.
"Error in the code"
The code is written with an error and does something different from the programmer’s intent.
"Integration errors"
Two programmers wrote interacting modules / classes, and misunderstood the protocol. A protocol means both parameters passed to functions and a valid / valid sequence of calls.
Moreover, each of the classes can be written “purely”, but when combined, they give a teravilny system operation.
"Multisulting and Timing Errors"
If several threads participate in the work, there may be errors associated with access to a shared resource or with a sequence of calls. These are the most elusive (after memory corruption) bugs that I have encountered.
“UI Errors” and “Unexpected Errors”
The UI does not meet the requirements at all or somehow crookedly handles some specific situations.
For example, it allows for the input of incorrect data or does not display a sufficiently clear message in case of a negative result.
Pros and cons of various types of testing
| Manual | Functional | Unit | |
|---|---|---|---|
| Ability to find "errors in the code" | Yes | Yes | Yes |
| Ability to find "integration errors" | Yes | Yes | Not |
| Ability to find "Multisulting errors and timing" | Partially | Partially | Not |
| Ability to find “UI errors” and “contingency errors” | Yes | Partially | Not |
| Ability to test with a variety of input data | Low | Very high | |
| Ability to automate testing | Very low | Low | Yes |
| Extra programming effort | Not | No ... x1.5 | x2 ... x5 |
According to my experience (I don’t have accurate statistics, unfortunately), “errors in the code” make up less than 30% of all bugs in a fairly complex system. If the programmer is experienced enough, the number of such errors will be even less.
The vast majority of such errors are perfectly captured using “code reviews” (code review) and functional tests.
Unit tests are capable of testing code with a wide variety of possible options, up to all conceivable input data. You can test the code very carefully, you can even "to the last comma."
The problem is that there is a certain set of situations that can happen in a real system. Testing something beyond this set is irrelevant and is a waste of time.
Typically, functional tests cover a completely rational set of possible situations.
The strongest side of unit tests is automation.
But personally, I (over 3 years of work) did not come across a situation where unit tests helped find a bug quite a long time after it was written.
That is, unit tests are effective (helping to find bugs) for an exceptionally short time after writing.
I have never seen any unit tests run automatically on the builder unit. They are there "clean clean." And, of course, they serve the guidance of peace of mind. :)
Minus unit tests are a huge effort to spend on writing and programming mocks. That is why the time required to develop using unit tests is 2-5 times longer than without them.
Of course, if the code weakly uses other modules, so that it is very easy to isolate it completely, then the unit tests almost intersect with the functional tests and the difference between them is only in the name and minor technical details.
Summary:
Unit tests do not give a significant gain in system reliability.
There are 3 types of code when unit tests are very appropriate and their use is justified:
- High reliability software - no expense here
- Easily isolated code - writing unit tests is easy, simple, and the cost of them is no more than that of functional tests.
- Utilities is a code that will be VERY widely used in the system, so it’s worth thoroughly scanning them in advance
If the cost of the error is exceptionally high (for example, we write the software of an airplane or a power plant), then, of course, no costs are excessive and unit tests must be used. But usually the cost of the error is much less.
Organizational and psychological issues
The backfill question is: who and how can make sure that the programmer wrote unit tests well enough?
There are two possible options for testing unit tests:
Instrumental check
It is possible to measure the "coverage" of the code with tests - whether all the lines of code were executed as part of tests. I have never seen such a check used. Well, of course, such tests can verify that the tests "cover" the code, but they cannot verify that the tests "cover" the possible input data.
Inspection of unit tests by another programmer or supervisor
I have seen it done. Writing unit tests is a very boring thing. Understanding them is generally dark. That is, you can understand, but it requires a serious amount of time and motivation. At the very beginning of implementation, people had an interest and motivation to do everything as it should to figure out what it was - unit tests and TDD. But it passed very soon.
Thus, it is quite difficult to control the programmer that he wrote unit tests well.
As soon as control and administrative pressure decreases, programmers, here are lazy people, tend to write unit tests more and more “for show” and test their code using manual or functional testing.
For systems with a not too high error cost, programmers do not have enough pressure to be paranoid for honest use of unit tests.
If the programmer misses something in the code, it is caught in QA or already at the client and the bug is returned for correction. Feedback makes you take the code seriously and attentively.
If a programmer misses something in writing tests, in fact, no one will ever notice. The lack of feedback allows you to treat the question of disregard.
Even if the programmer got five times more time to develop,
How else can you make unit tests "work"? They should work if the tests are written by another programmer, and it’s best to the code itself.
And here we come to TDD.
Test Driven Development
If the test is written before and provided to the programmer as source data, it becomes a “technical requirement” (requirement). Of course, the one who writes the test must write and program all the required mocks in advance. Haha, I would not want to be the programmer whose responsibility it is to program tests for others.
Well, can a programmer himself write tests to his code in advance?
In my experience, no and no. Programmers (and I and the vast majority with whom I discussed it) just think in the opposite direction.
Yes, at first people tried to honestly follow the methodology. But after some time, they started to write the code first and then add tests to it. And after some time, the tests lost their versatility and meticulousness. And then “shtob was” was written at all, since they ask.
Does it even need it?
Source: https://habr.com/ru/post/207448/
All Articles