We keep the design of the system under control using isolated unit testing
Agree, the situation when we want to throw out a bunch of ready-made code is very annoying. In this article, together with Andrey Kolomensky, we will try to figure out what the reasons may be for this, and how to find out how our system should look like at the point of maximum productivity. Let us consider which approach will drag us into a vicious circle of insufficiently careful design, and which will allow us to obtain a testable system, which ultimately leads to a high-quality system design and reduces the risk of defects.
Today we talk about ')
How to make testing complex dependencies?
How to achieve a large test coverage?
How do tests affect design?
What to do when a lot of logic in the database?
How to keep a compromise between design and "not design".
About the speaker: Andrei Kolomensky - Agile Coach at OnAgile , has been writing code for over 10 years, working on complex domain models, such as payment systems, and on developing complex legacy codes when they had to be rescued and restored to work on them. .
For all the time of my work, I noticed the following problem. When we are just starting a project or are already working on it, we always expect that we will move at a constant speed (as in the title picture). But in reality, reality does not agree with us.
All of us very often slide into wild unproductiveness. At the beginning, the business is happy that we, as programmers, implement a lot of features, and in the end we complain that we supply few features. As a result, in the middle zone, approximately where the question mark is placed, we have a strong desire to rewrite everything or conduct a major refactoring.
The situation when we want to throw out the code annoys me greatly. This is a very close topic to me. I developed a bank wallet solution for a year and a half. All this time we did not go into production, and when we almost had everything ready, the license was revoked from the bank.
The business decides not to drop the code and make another product: a payment aggregator, instead of a wallet solution. The subject area is very similar: we take money from users, pick up a commission, give money to the store.
We threw out all our code, because we could not make such a pivot , even in a similar subject area. There were several reasons for this.
Our code was too hard. Rigidity ( rigidity) says that the system resists making changes. To make changes to the system, we need to touch on a lot of components.
Our system was fragile. This is the tendency of the system to break down in a variety of places when making seemingly small changes.
Our system was intolerable - immobility . This is the quality of the system, which suggests that we cannot reuse code from one system to another. More precisely, we can, but the cost of extraction will be more expensive than the cost of writing code from the very beginning.
Our system contained unnecessary duplications ( needless repetition ) and excessive complexity ( needless complexity ).
The clarity ( opacity ) of the expression of intentions was rather low, even though we focused on it. A common mistake of programmers is that when we launch a new product and do not go into production for a long time, we make the groundwork for the future in order to save. Because of this, we did not understand exactly how those parts of the system that were stubs should be arranged, what exactly those elements should represent from themselves, and what behaviors and dependencies should have. The dysfunctions I listed above prevented clarity from reaching the rest of the system.
The last parameter viscosity I rendered separately. This is an attribute of quality, which indicates how much the system resists the application of high-quality architectural solutions. For example, if the tests pass an hour and there is no talk about TDD, this is a system with a huge viscosity.
Question: how to find out how our system should look like at the point of maximum productivity?
The code we threw was covered with tests. We took care of its quality, refactored, but as a result, we threw out almost all.
We need some kind of tool to get feedback from the system, which would help us to understand how we should design our system. The knowledge that we have in our head may not be enough to know at any given moment how our system should look like in general.
Unit tests - the main tool for getting feedback from the system
When we write unit tests, we can at least guarantee the correctness of the system and some small quality of its testability.
Let's look at one test.
There is an example in vacuum BuyProductsAction - we buy some products. I have questions to this test, the main one being: what can I learn about the quality of the system from this test? Virtually nothing: I can iterate through the input parameters, add more asserts, somehow provide additional checks. Moreover, you need to check quite a lot, because we have too many product and user characteristics and too many parameters in the database.
I don’t know anything about system design here. This is the reason why we threw out all the code - because what we learned from our tests did not allow us to reasonably improve the design of our system.
What can BuyProductsAction do within itself? Create an order, send notifications, debit money from the account, charge interest bonuses - he can do a lot within himself.
What is an integrated test?
I’ll go away from the concept of unit test, because it’s too vague, everyone understands it in his own way.
Integrated tests are tests that pass or drop. depends on more than one unit of nontrivial behavior.
That is, we can not specifically show the point why this test fell. When we see that there is an error somewhere, that some component from the testing area has fallen, and not a specific one place, then this test is integrated.
Isolated tests are tests for one non-trivial unit. behavior, the passage or fall of which depends only on this behavior.
In fact, this is a test for one method or for one section of the system, and the rest of the non-trivial behavior is replaced by moki.
Suppose that we do, if we try to make the test for the BuyProductsAction method isolated, so as not to test everything entirely, but run only the run method, which will be isolated from any nontrivial dependency behavior it contains.
Most likely, we will not be able to do this, because systems that are written with such tests do not have such a strong impact on the system design so that we can immediately write isolated tests. Even if we can do it, most likely there will be trash:
We begin by saying that we have some kind of AnaliticsComponent, where the input parameters are passed. We thrust this business into a service locator. We still have some components whose behavior we are asking.
More components that do not need to be read are simply to make the volume clear.
The number of dependencies that we see in an isolated test, when we explicitly prescribe them, is usually quite large if there is no practice of writing isolated tests in a company. Even if we can write an isolated test, the situation usually looks like this.
The first thing I do when I refactor the system into a testable state, I start cramming dependencies into classes and explicitly injecting them into the constructor.
The main question you can ask when looking at this test is what do I learn about the quality of the system from this test?
I see that I have clearly violated the principle of sole responsibility. There is no longer a subjective argument about "clarity." I see that this test is hard for me to read and write. I see that this test will constantly fall, because any change I have will be made in this class. This fictional test was hard for me even to prepare for the presentation.
If we wrote this production code, we would just go crazy.
I use isolated tests to improve the quality of the system design and its reasonable refactoring.
They give the most complete feedback about my system.
If the integrated tests give me practically nothing but a basic understanding that part of my system works correctly on the part of the input parameters, then isolated tests allow me to clearly see what is happening in my system. The degree of discomfort with which I write isolated tests will correspond to the degree of testability of my system.
System rotting
We passed all unit tests, but the QA department found some kind of defect. We understand that the problem is at the junction of two components, and we decide to write an integrated test, because it’s easier and because we have to check the real work - how our system works, because we still had a bug.
By the way, I do not recommend using the word bug . This is a fly that flew into servers at the dawn of our industry. An example of a bug from IT development is when I copied a SQL request from Skype, pasted it into the code, and it does not work there, because Skype instead of a space inserted non-breaking space. When this happens, this is a bug. In the rest I prefer to use the word defect in cases as an incorrect program behavior is not an accident, but the direct responsibility of programmers. Defect much more powerful wording than removing the responsibility "bug." There are no specific proofs, but one the team managed simply by passing from the word bug to the word defect, to increase the quality, simply by increasing awareness and responsibility.
Since we wrote an integrated test, it has less impact on the design of our system. When we write an integrated test, we can write its implementation in different ways: insert a bunch of dependencies, make a call to static methods that change the behavior of the system, call from the service locator just a shroud of calls — absolutely anything — we have complete freedom.
Therefore, from the easy life, we begin to design the system less carefully. We do not care how our system will be arranged - only on our own sense of inner beauty, we look at the system and think how best. But no pressure from the test occurs.
This leads to the fact that the testability of our system decreases, and now we can not write a small isolated test. At least, even if we can write, it has become harder to do.
In this regard, we have a greater risk of defects, because testing our system becomes more difficult. We have less time to write quality small isolated unit tests.
We return to the circle and eventually come to the decision to write only integrated tests, because isolated tests are difficult to write. As a result, we get a situation where nothing influences the design of our system, only we, as we want, or write.
Alternative option.
The same situation - the unit passed the test, but there was a defect. What will happen if we write a small isolated test. We will face a bunch of problems with the fact that the system prevents us from writing these small isolated tests.
In order to make life easier and just start writing these isolated tests with high quality, so that we understand what is happening there, we start designing the system more carefully so that we don’t have to write huge tests.
In order for the tests to be small, you need to try very hard to make the system high-quality and consistent with the principle of sole responsibility. This leads to the fact that the testability of our system increases. We try to make our system as testable as possible so that we can write isolated unit tests more easily.
As a result, with a testable system, we have more time to write small isolated unit tests, which leads to a high-quality system design and reduces the risk of defects. What about not doing “real work”? Before I cover this topic, I want to touch on one more thing. My point is that:
Continuing to maintain high productivity is only possible. practicing the discipline of test driven development.
Although the opposite opinion is also quite common (for example, a video on this topic).
Test Driven Development
Test Driven Development is a discipline. Discipline implies a limitation that we impose upon ourselves by applying it. This is not a Red-Green-Refactoring, but a series of specific rules:
As long as you do not have a falling unit test, you cannot write production code.
You are forbidden to write more unit test code than enough to drop it. Any compilation error is a crash. You immediately stop writing a unit test as soon as it crashes, even with a compilation error.
You are forbidden to write more production code than it is enough to pass one falling unit test, and you cannot write that production code that does not relate to a specific falling unit test.
We do not just write a test, and then the implementation is not just Red-Green-Refactoring, these are additional restrictions. This - Test Driven Development - that allows you to maintain our system in a qualitative state and maintain the highest possible productivity over a long period of time.
Legacy code
According to the definition given by Michaels C. Feathers, a legacy code is a code without unit tests . It's simple. In my practice, I notice a direct correlation between the lack of tests and the presence of a huge number of problems with the design of the system, as well as the presence of integrated tests and about the same dependence with the problems of system design. The less small isolated tests, the more problems with the design of the system.
When there are no tests, it is legacy.
I like another definition that is less accurate, but reflects reality.
Legacy code is a code that is scary to change.
Dave Thomas once said something like: “In some cases, I don’t write tests at all; I can design a good quality system well anyway.”
First of all, he has a huge amount of unit testing experience. Secondly, when I work with such a system without tests, for me this system will be legacy, because it will be scary for me to make changes to it. Fear of making changes is the main cause of code rot. Discipline Test Driven Development - medicine. With a committed use of this discipline, the fear of making a change goes away, since you get feedback on every line of your code.
Robert Martin proposes that in our profession we should take an oath similar to the Hippocratic Oath for doctors. I bring it here to clearly demonstrate why unit tests are at least important for our industry, and, as a maximum, Test Driven Development, as a discipline, important for us as programmers.
Oath programmer
In order to protect and preserve the honor of the profession of programmers, I promise that to the best of my abilities and judgment:
1. I will not create malicious code.
This applies not only to viruses, but also to the code that creates losses for our company. If we wrote the code that caused the company losses - this is malicious code.
2. The code that I create will always be my best work.
I will not consciously allow my code to be defective, both in behavior and in structure. In the behavior, of course, we can not guarantee its correctness, if we do not have some kind of verification. In the structure, if there are no small isolated unit tests, we cannot guarantee that the design of our system is testable and of high quality.
3. I will provide with each release a quick, reliable, and repeatable proof that each element of the code works as it should.
The importance of this item is especially noticeable if you see how much money many companies spend on manual testing, something that can be automated and used at the same time to improve the quality of the code, and as a result, to speed up development.
4. I will make frequent, small releases so as not to interfere with the progress of others.
In order to deliver quickly, we need to make small releases. I personally cannot do small releases with a sufficient degree of quality without tests.
5. I will fearlessly and tirelessly improve my code at every opportunity.I will never reduce its quality.
In order to refactor the code, we need to not be afraid to change it. My old pattern looked like this: I see a place in the system and 2 ways to make changes to this system: an easy way (“crutch”) and a complicated way when I need to do a serious refactoring.
If I do not have tests, I will not seriously change the structure of the subject area, because I can break something. And I do not want to break something, so I choose the easy way.
With unit tests, I have no such problem. While practicing Test Driven Development, there are no problems at all - my code is always correct. If something breaks there, it is a pain for me as a professional, because I feel that I screwed up somewhere, because the situation was under my complete control. For me personally, the occurrence of defects is a serious challenge.
6. I will do everything I can in order to maintain the productivity of myself and others as high as possible.I will not do anything that reduces this productivity
It is often said that Test Driven Development reduces productivity, or increases it when we practice it for a long time. In fact, it keeps productivity at a constant level.
Our task is not to move faster, our task is to move at a constant speed and to make sure that we get rid of the losses associated with weak architectural solutions.
Test Driven Development allows us to do this.
7. I will constantly make sure that others can replace me, and I could replace them
If I see the code of another person and there are no tests there, this is a problem for me. I can figure out what is happening there, but I am afraid to make changes there, since I can not take something into account. If our team practices pair programming and full coverage with unit tests, it’s very easy for us to replace each other and work on different parts of the system - everything is safe.
8. I will give assessments that are honest, both in their correctness and accuracy.I will not make promises without the confidence that I can keep them
If we have a terrible law, and we say that it will take a week, and then we dig out a place with bad code, this week turns into three - a very frequent phenomenon with legacy code.
9. I will never stop learning my craft.
Little exercise
I suggest you check the statement that Test Driven Development allows you to keep productivity at a maximum constant level. I want you to try to find in your product a part of the system that you consider well designed, for which, perhaps, there are tests, but they are integrated, and try to write a small isolated unit test for this part of the system. Then I will show how I write them personally.
The degree of discomfort you will experience is consistent with the quality level of this system. A small isolated test will give you an understanding of how well or poorly designed your system is.
If you realize that the system is not very well designed, this is a reason to start thinking about using Test Driven Development.
Simplest example
There is a client that depends on the server. The client is context independent, testing it is very easy. We simply call the methods and look at the output. It is more difficult to test the server, now it is tied with nails to the client. In order to test it independently, we need to separate them.
We have to insert the interface in the middle. Now we can test the server, regardless of the client . Probably, many have heard the advice of programming on the basis of the interface, and not the implementation, but did not understand why this advice is good.
This is a demonstration of why this is so. If we want to write small isolated unit tests that help us with the design of our system, we must somehow separate our components. In order to separate them, we need to insert something in the middle. In this case, it is an interface.
An interface is not just a set of methods and signatures for incoming and outgoing parameters. The interface is also a contract, that is, the client's expectation, which must be satisfied when he asks something from the interface.
Suppose we ask the interface to return active users. It is not enough just to check that we have an array of users in the outgoing parameter. We need to be active. Therefore, we write a test in which we ask: "Interface, please give us active users!"
And we imitate real work - because we don’t need to do real work, we write an isolated test.
Stub-ohm return some value:
An empty array means no users.
1 user - immediately enter into the profile of one user.
If there are many users, we display a list of tables.
All - we wrote three tests. Now our task is to make the contract run in some kind of implementation. We really climb into the database, we get something and check that we really get active users. We act in accordance with the contract interface.
Thus, on the left, we specifically ask how our system behaves with a different result of the interface, on the right - that the system actually fulfills the expectations, as a result we get that everything works together.
We do not need to test the client and server together in order to verify their correctness. It is enough that we:
we ask correctly
process correctly
we act correctly
check correctly.
It's enough.
The 4 Rules of Simple Design
This concept came up with Kent Beck. In order of priority, a system can be considered simple if it passes all tests.
If there are no tests in the system or the code does not pass the tests, it cannot be considered simple, at least, because the viscosity of the system is very high. This code is scary to change and this is a problem because the system will start to rot. As a result, productivity will fall.
Tests - a prerequisite for the code to be considered simple.
Then we can concentrate on clarifying our intentions. I do not remember who said that the code should be read as well-written prose. After all the tests work for us, we can take care that our code is read as well-written prose and remove unnecessary duplication .
At the very end, we can already think about the fact that our system consists of the smallest number of elements .
I advise you to add these guys as friends in the social. networks:
Kent Beck is the founder of the extreme programming movement and the Test Driven Development discipline.
J. B. Rainsberger - founder of JetBrains.
Robert Martin - I am sure you know about him, if not - give maximum priority to reading his blog and books.
Subscribe to them, take an example from them, read blogs, study what they write.
What's next?
I want to challenge you. Returning to a small exercise: try to take the part of the system that you consider well designed, and write a small isolated unit test on it.
Most likely, you will encounter the fact that you cannot do this, and if you decide to do something with it, I advise you to look at these resources:
Thank you for your commitment to reading this article.
You can write to Andrew in a telegram https://t.me/akolomensky to ask questions or ask for advice on engineering, process or product cases, he promised to answer everyone, so do not hesitate.
We want to note, isn’t it in order to comply with the clause of the oath “ I will never stop learning my craft ”, and continuously improve myself , we meet at conferences. After all, the intensive flow of ideas and cases from practice, received at conferences, gives impetus to self-improvement for at least six months. And so that the development schedule does not look like a bad production schedule, it’s time to get a new charge - the RIT ++ festival will be on May 28 and 29 , and Highload ++ Siberiaon June 25 and 26 in Novosibirsk . On the last until April 30, you can have time to submit applications.