Unit tests: what, how and when to test?
Testing software code is a painstaking and complicated process. The lion's share of work in it is done by unit tests. Until they “light up green”, there is no sense to test further.
How to write unit-tests correctly? Should I chase the 100% coverage? What difficulties do engineers face in practice? Marc Philipp and Vsevolod Brekelov share their experiences.
Marc Philipp is one of the main developers of the JUnit 5 framework - a tool for Java testers. Currently working as an engineer in the German company LogMeIn on cloud SaaS-solutions.
')
Vsevolod Brekelov - Senior QA Engineer in the company Grid Dynamics, has been testing for more than 5 years, has experience in building test automation from scratch.
- In articles about unit testing, examples of calculator methods and classes are usually given as examples. Such examples can show the complexity of real problems? What does a tester of full-featured programs face?
Marc Philipp: Indeed, with examples with a calculator, it is impossible to show the complexity of real tasks. They are selected in articles to enable readers to focus on understanding unit-testing approaches without the need to parse complex code. Although these examples are very simple, they demonstrate well the basic idea and principles of unit testing. In real life, the code under test should be originally written taking into account the fact that Unit testing will be carried out on it. One way to ensure this is to write tests before writing the code or almost simultaneously with it. When you have code adapted for testing, writing unit tests is not much more difficult than for a calculator.
Vsevolod Brekelov: I think that the complexity of real tasks can be understood only on real tasks. Seriously, there are some good articles where nontrivial examples are considered in great detail. I think that they will help to approach in reality.
For example, on request “unit testing java” you can quickly find an article on Habré . It was published for a long time, but has not lost its relevance.
As for the features of the work, I would single out the following groups of testers (I hope not to offend anyone):
These people usually do not only test the software itself, but also the requirements and the process itself. True, they approach this more often formally, which, in my opinion, is wrong.
I would like to draw attention to the process. I believe that every tester should be well versed in building the development process, as in my practice, the legs, bugs, and the main waste of time on the implementation of what is not needed grow right from there.
- Each test should check one thing. How fully in practice can this condition be met? How do you fight addictions, what frameworks do you use?
Marc Philipp: When writing unit tests, usually one sample of the input data from the equivalence class in the tested problem area is taken. Of course, you must first define these equivalence classes. In each test, you add assertion only for those properties that are relevant to your test. You should not copy the same assertions into each new test and drive them away. When you have dependencies that affect the operation of a unit, consider using stubs or mocks to keep the test independent.
Many of our unit tests for JUnit 5 use mocks created by the mocking framework (the Mockito in our case). As I said above, they are very useful for testing isolated code. The main task here is to make sure that your mox behaves like a real code. Otherwise, the tests will become meaningless.
Vsevolod Brekelov: Yes, there is an opinion: one unit test - one assertion. In practice, I have seen this very rarely. I think that this is the philosophy of the team. Multiple assertions are quite self-occurring.
If we conduct unit tests, not component tests, we isolate all dependencies (mocks, stubs - everything is in your hands). There are no difficulties in my opinion. And if they do, StackOverflow will definitely help.
As I write in Java / JavaScript (Angular), I use the usual popular tools:
in Java - Mockito / EasyMock. For component tests, writing your responsive mock is also a good idea! I advise everyone.
JavaScript - ngMock. By the way, for component tests a very cool topic - AngularPlayground .
- How to find a compromise between the labor and financial costs of testing and the quality of the final software in the implementation of "burning" projects? How do you usually argue the importance of full testing in such cases?
Marc Philipp : In my experience, you cannot save a “burning” project by skipping tests. Writing unit tests is an integral part of software development. Without it, you will not be able to find out if your code really does what you think it should do. You will not be able to quickly fix anything, because you will not understand where that has broken. As UncleBob said , “the only way to go quickly is to go well.”
Vsevolod Brekelov: I think there is no definite answer. Rather, it helps experience and type of project. If you are doing a medical project or building a rocket, then the importance of testing does not have to say. If you saw a startup in a week - what tests?
It is very important to organize the process to avoid sudden bugs and incorrectly implemented requirements. What is the correct process? Of course, there is Agile Manifesto, which many people look at when organizing the process, but still something does not work. You can take and build a process for the sake of the process. And you can, and vice versa, follow http://programming-motherfucker.com/ .
It seems to me that the main thing is to have requirements, the details of which suit developers and testers in a team. This means that they have the same understanding of what will be output.
- What techniques help reduce the time and labor costs for testing?
Marc Philipp: “Testing” is an overloaded term. It can mean anything: unit testing, manual testing, performance testing ... In my experience, manual testing, that is, manual execution of the test case walkthrough plan, is really expensive and often not as effective as you think. Moreover, automating these boring tests only makes sense to a certain extent. However, you should really follow the test pyramid , and not write too many of these end-to-end / UI tests. Most of your tests should be real unit tests: independent, fast tests that you can perform very often. Writing these tests is relatively cheap, especially if you know your tools. They are very reliable, so you will not waste time updating them. UI and Integration tests will always be more fragile due to the sheer number of components involved.
Vsevolod Brekelov: There is a good technique - write less code.
The main thing is to understand the process and what you want to decide (or test).
Always need to adequately assess the budget and time. What does it mean? If you can afford to pour a lot of money into approaching 100% coverage - why not? The owner is the master.
If you do not have money for autotests (which, as you know, are fighting off mainly in long-running projects), then a crowd of manual testers is your option.
If you don’t go to extremes, the most common mistake is writing e2e tests in batches before losing a pulse before unit tests, component tests, integration tests for Backend, Frontend, DB, Performance, etc. are written. This trend probably follows the trendy BDD approaches (I don’t really like them). What does all this lead to?
The first degree of "intoxication" - automation starts to work for you. You replace manual test cases with automatic ones. Testers are starting to rejoice. Managers are starting to think they are about to save.
Second degree - there are a lot of tests; for some reason, some of them periodically fall. Testers are not very happy. Need to sit and understand the reasons. But bugs still get through. And, probably, they are even on QA environment by manual (maybe even monkey) testing.
Third degree - everyone starts to go to conferences about Selenium (I have nothing against these conferences), learn how to deal with Flaky tests, try different solutions. Let tests in parallel.
The fourth degree is to build the entire hierarchy of the launch of 500 e2e tests on 50 agents, so that everything can fly fast, in as little as 10 minutes (I'm exaggerating here, of course). And still there are bugs.
Fifth degree - I will call it unreachable. There comes the realization that most of the e2e tests are not needed. Looking for other tests that no one ever wrote. For example, component tests on the back-end or the same on the UI. Or maybe not they, maybe system tests? And maybe the layout tests? Or maybe your {username} option?
Certainly there are projects where everything is done "correctly." But often there is the problem of misunderstanding what needs to be tested. Only proper understanding can save your time and finances. And what's more, improve the quality of the product.
- How does the development of development tools and code creation approaches affect testers tools and approaches? What innovations make it easier
unit-testing (for example, the presentation of methods in the form of lambda functions)?
Marc Philipp: New tools are trying to make life easier for developers, giving them more flexibility. However, in the end, I believe that it does not matter if you present your tests as methods or as lambda functions. Understanding what to test and how to test is the hardest part.
Vsevolod Brekelov: The development of tools and approaches affects positively if they are used. It is not always possible to apply HYIP technologies or approaches at work. We still solve business problems. But you can always find a balance.
Which makes testing easier is a strange question. I think that technology can not greatly facilitate life. Since, in order to use something new (technology, tool), it needs to be studied by the whole team, by adopting some kind of “policy”, code style. This in the future can, of course, make life easier, but at short distances it is not very useful, as it is labor-intensive, IMHO.
By the way, the option of switching to Kotlin (if we are talking about Java tests) may be a good idea. I have not tried it in my practice yet.
Regarding the innovations of the language (lambdas and other utilities) - this is all well and good, of course, but it is difficult for me to say how much they make life easier, as it should be measured. I did not measure. But just do not write me down as opponents of progress, I believe that the practice of learning / using something new should always be present. This is the usual continuos improvement story.
- How much do you cover unit-tests your production projects? Is it worth spending time on 100% coverage?
Marc Philipp: Depending on the programming language and the frameworks you use, there may be some template code in the project that does not contain any logic. But apart from such pieces, in my opinion, you should write unit tests for all your code. Thus, I would recommend coverage of more than 90%.
Vsevolod Brekelov: In projects in which I had to work, the developers most often try to bring the tests to a coverage of 90%. Is it worth spending time - usually decided by managers. I am not a manager, but unit tests are a very good practice for me, it’s good to have 100% coverage when there are resources for it.
The main thing is to remember that 100% coverage, unfortunately, does not guarantee that you do not have bugs.
From what seems more useful than the race from 90% to 100% coverage, this is writing mutation tests. I will not say anything new about the 2012 article . But in practice, I did not often see that this approach was applied (and I myself, too, repent). So maybe it's time to start?
- How do test frameworks help with unit tests? What part of the work do they take on? What not to expect when using frameworks?
Marc Philipp: A good framework makes it very quick and easy to write simple unit tests and at the same time contain powerful mechanisms for conducting more complex tests. For example, it should help you prepare test data and provide extension points that allow you to reuse the same logic in many tests. But no framework decides for you what to do and how to test it. Also, he cannot magically improve your project to make it well testable.
- Which elements of the code are the most difficult to unit test? How do you solve this problem with you?
Vsevolod Brekelov: The more dependencies - the more routine, the harder it is to write a unit test. But in general, I do not see any particular problems, to be honest. Although a large number of books have been written on the subject of unit tests, of which I have not read one to the end. Maybe that's why I'm not burdened with problems.
For example, it is difficult to write a unit-test, when, say, the object constructor contains code vermicels, but then you can advise comrades to read books,
for example , enter code review practice.
As for JavaScript code, there you can meet with various difficulties and surprises (yes, I really like JavaScript), rather related to the framework used, for example, working with digest. I used only AngularJS / Angular2 / Angular4. Despite the efforts of the Angular team to make a conveniently-tested framework, you still occasionally encounter problems that certainly have solutions, because we are engineers.
A huge array of information about all aspects of testing is waiting for participants at the nearest Heisenbag, where Mark Phillip will read the report "JUnit 5 - The New Testing Framework for Java and Platform for the JVM".
You can find out what other significant figures will perform at the conference and be able to answer the most pressing questions on the sidelines on the site .
How to write unit-tests correctly? Should I chase the 100% coverage? What difficulties do engineers face in practice? Marc Philipp and Vsevolod Brekelov share their experiences.
Marc Philipp is one of the main developers of the JUnit 5 framework - a tool for Java testers. Currently working as an engineer in the German company LogMeIn on cloud SaaS-solutions.')
Vsevolod Brekelov - Senior QA Engineer in the company Grid Dynamics, has been testing for more than 5 years, has experience in building test automation from scratch.- In articles about unit testing, examples of calculator methods and classes are usually given as examples. Such examples can show the complexity of real problems? What does a tester of full-featured programs face?
Marc Philipp: Indeed, with examples with a calculator, it is impossible to show the complexity of real tasks. They are selected in articles to enable readers to focus on understanding unit-testing approaches without the need to parse complex code. Although these examples are very simple, they demonstrate well the basic idea and principles of unit testing. In real life, the code under test should be originally written taking into account the fact that Unit testing will be carried out on it. One way to ensure this is to write tests before writing the code or almost simultaneously with it. When you have code adapted for testing, writing unit tests is not much more difficult than for a calculator.
Vsevolod Brekelov: I think that the complexity of real tasks can be understood only on real tasks. Seriously, there are some good articles where nontrivial examples are considered in great detail. I think that they will help to approach in reality.
For example, on request “unit testing java” you can quickly find an article on Habré . It was published for a long time, but has not lost its relevance.
As for the features of the work, I would single out the following groups of testers (I hope not to offend anyone):
- Back-end - can write system, integration, component, unit tests (why not?).
- Front-end - can write both e2e tests, and component, and unit tests.
- DB - are engaged in testing data / the database itself.
- Performance - here it seems to be obvious.
- Infrastructure - deal more with “near-loop” issues.
- Mobile testing (iOS, Androind, IoT) - now it has become very fashionable to separate such engineers, although in my opinion it's all about the same Back-end / Front-end.
These people usually do not only test the software itself, but also the requirements and the process itself. True, they approach this more often formally, which, in my opinion, is wrong.
I would like to draw attention to the process. I believe that every tester should be well versed in building the development process, as in my practice, the legs, bugs, and the main waste of time on the implementation of what is not needed grow right from there.
- Each test should check one thing. How fully in practice can this condition be met? How do you fight addictions, what frameworks do you use?
Marc Philipp: When writing unit tests, usually one sample of the input data from the equivalence class in the tested problem area is taken. Of course, you must first define these equivalence classes. In each test, you add assertion only for those properties that are relevant to your test. You should not copy the same assertions into each new test and drive them away. When you have dependencies that affect the operation of a unit, consider using stubs or mocks to keep the test independent.
Many of our unit tests for JUnit 5 use mocks created by the mocking framework (the Mockito in our case). As I said above, they are very useful for testing isolated code. The main task here is to make sure that your mox behaves like a real code. Otherwise, the tests will become meaningless.
Vsevolod Brekelov: Yes, there is an opinion: one unit test - one assertion. In practice, I have seen this very rarely. I think that this is the philosophy of the team. Multiple assertions are quite self-occurring.
If we conduct unit tests, not component tests, we isolate all dependencies (mocks, stubs - everything is in your hands). There are no difficulties in my opinion. And if they do, StackOverflow will definitely help.
As I write in Java / JavaScript (Angular), I use the usual popular tools:
in Java - Mockito / EasyMock. For component tests, writing your responsive mock is also a good idea! I advise everyone.
JavaScript - ngMock. By the way, for component tests a very cool topic - AngularPlayground .
- How to find a compromise between the labor and financial costs of testing and the quality of the final software in the implementation of "burning" projects? How do you usually argue the importance of full testing in such cases?
Marc Philipp : In my experience, you cannot save a “burning” project by skipping tests. Writing unit tests is an integral part of software development. Without it, you will not be able to find out if your code really does what you think it should do. You will not be able to quickly fix anything, because you will not understand where that has broken. As UncleBob said , “the only way to go quickly is to go well.”
Vsevolod Brekelov: I think there is no definite answer. Rather, it helps experience and type of project. If you are doing a medical project or building a rocket, then the importance of testing does not have to say. If you saw a startup in a week - what tests?
It is very important to organize the process to avoid sudden bugs and incorrectly implemented requirements. What is the correct process? Of course, there is Agile Manifesto, which many people look at when organizing the process, but still something does not work. You can take and build a process for the sake of the process. And you can, and vice versa, follow http://programming-motherfucker.com/ .
It seems to me that the main thing is to have requirements, the details of which suit developers and testers in a team. This means that they have the same understanding of what will be output.
- What techniques help reduce the time and labor costs for testing?
Marc Philipp: “Testing” is an overloaded term. It can mean anything: unit testing, manual testing, performance testing ... In my experience, manual testing, that is, manual execution of the test case walkthrough plan, is really expensive and often not as effective as you think. Moreover, automating these boring tests only makes sense to a certain extent. However, you should really follow the test pyramid , and not write too many of these end-to-end / UI tests. Most of your tests should be real unit tests: independent, fast tests that you can perform very often. Writing these tests is relatively cheap, especially if you know your tools. They are very reliable, so you will not waste time updating them. UI and Integration tests will always be more fragile due to the sheer number of components involved.
Vsevolod Brekelov: There is a good technique - write less code.
The main thing is to understand the process and what you want to decide (or test).
Always need to adequately assess the budget and time. What does it mean? If you can afford to pour a lot of money into approaching 100% coverage - why not? The owner is the master.
If you do not have money for autotests (which, as you know, are fighting off mainly in long-running projects), then a crowd of manual testers is your option.
If you don’t go to extremes, the most common mistake is writing e2e tests in batches before losing a pulse before unit tests, component tests, integration tests for Backend, Frontend, DB, Performance, etc. are written. This trend probably follows the trendy BDD approaches (I don’t really like them). What does all this lead to?
The first degree of "intoxication" - automation starts to work for you. You replace manual test cases with automatic ones. Testers are starting to rejoice. Managers are starting to think they are about to save.
Second degree - there are a lot of tests; for some reason, some of them periodically fall. Testers are not very happy. Need to sit and understand the reasons. But bugs still get through. And, probably, they are even on QA environment by manual (maybe even monkey) testing.
Third degree - everyone starts to go to conferences about Selenium (I have nothing against these conferences), learn how to deal with Flaky tests, try different solutions. Let tests in parallel.
The fourth degree is to build the entire hierarchy of the launch of 500 e2e tests on 50 agents, so that everything can fly fast, in as little as 10 minutes (I'm exaggerating here, of course). And still there are bugs.
Fifth degree - I will call it unreachable. There comes the realization that most of the e2e tests are not needed. Looking for other tests that no one ever wrote. For example, component tests on the back-end or the same on the UI. Or maybe not they, maybe system tests? And maybe the layout tests? Or maybe your {username} option?
Certainly there are projects where everything is done "correctly." But often there is the problem of misunderstanding what needs to be tested. Only proper understanding can save your time and finances. And what's more, improve the quality of the product.
- How does the development of development tools and code creation approaches affect testers tools and approaches? What innovations make it easier
unit-testing (for example, the presentation of methods in the form of lambda functions)?
Marc Philipp: New tools are trying to make life easier for developers, giving them more flexibility. However, in the end, I believe that it does not matter if you present your tests as methods or as lambda functions. Understanding what to test and how to test is the hardest part.
Vsevolod Brekelov: The development of tools and approaches affects positively if they are used. It is not always possible to apply HYIP technologies or approaches at work. We still solve business problems. But you can always find a balance.
Which makes testing easier is a strange question. I think that technology can not greatly facilitate life. Since, in order to use something new (technology, tool), it needs to be studied by the whole team, by adopting some kind of “policy”, code style. This in the future can, of course, make life easier, but at short distances it is not very useful, as it is labor-intensive, IMHO.
By the way, the option of switching to Kotlin (if we are talking about Java tests) may be a good idea. I have not tried it in my practice yet.
Regarding the innovations of the language (lambdas and other utilities) - this is all well and good, of course, but it is difficult for me to say how much they make life easier, as it should be measured. I did not measure. But just do not write me down as opponents of progress, I believe that the practice of learning / using something new should always be present. This is the usual continuos improvement story.
- How much do you cover unit-tests your production projects? Is it worth spending time on 100% coverage?
Marc Philipp: Depending on the programming language and the frameworks you use, there may be some template code in the project that does not contain any logic. But apart from such pieces, in my opinion, you should write unit tests for all your code. Thus, I would recommend coverage of more than 90%.
Vsevolod Brekelov: In projects in which I had to work, the developers most often try to bring the tests to a coverage of 90%. Is it worth spending time - usually decided by managers. I am not a manager, but unit tests are a very good practice for me, it’s good to have 100% coverage when there are resources for it.
The main thing is to remember that 100% coverage, unfortunately, does not guarantee that you do not have bugs.
From what seems more useful than the race from 90% to 100% coverage, this is writing mutation tests. I will not say anything new about the 2012 article . But in practice, I did not often see that this approach was applied (and I myself, too, repent). So maybe it's time to start?
- How do test frameworks help with unit tests? What part of the work do they take on? What not to expect when using frameworks?
Marc Philipp: A good framework makes it very quick and easy to write simple unit tests and at the same time contain powerful mechanisms for conducting more complex tests. For example, it should help you prepare test data and provide extension points that allow you to reuse the same logic in many tests. But no framework decides for you what to do and how to test it. Also, he cannot magically improve your project to make it well testable.
- Which elements of the code are the most difficult to unit test? How do you solve this problem with you?
Vsevolod Brekelov: The more dependencies - the more routine, the harder it is to write a unit test. But in general, I do not see any particular problems, to be honest. Although a large number of books have been written on the subject of unit tests, of which I have not read one to the end. Maybe that's why I'm not burdened with problems.
For example, it is difficult to write a unit-test, when, say, the object constructor contains code vermicels, but then you can advise comrades to read books,
for example , enter code review practice.
As for JavaScript code, there you can meet with various difficulties and surprises (yes, I really like JavaScript), rather related to the framework used, for example, working with digest. I used only AngularJS / Angular2 / Angular4. Despite the efforts of the Angular team to make a conveniently-tested framework, you still occasionally encounter problems that certainly have solutions, because we are engineers.
A huge array of information about all aspects of testing is waiting for participants at the nearest Heisenbag, where Mark Phillip will read the report "JUnit 5 - The New Testing Framework for Java and Platform for the JVM".
You can find out what other significant figures will perform at the conference and be able to answer the most pressing questions on the sidelines on the site .
Source: https://habr.com/ru/post/329372/
All Articles