How to stop worrying and start working?
Last time when we talked about the work of our team, many were interested in the details of organizing the work of the developers themselves, which we will now tell about. One should not expect “breaks in covers” and discoveries, because everything that developers do has been described and discussed more than once, but what we do together in real large projects is not done very often (to be honest, I don’t saw). That is, you should not expect something, but “cover breaks” will occur :)
The reality is that Agile without the correct engineering practices will end very quickly. If you do not make efforts that guarantee a high level of development quality and the state of the system as a whole, then as the project becomes more complex, control will be quickly lost. As a result, you will not be able to do everything planned in the iteration, and you will only dream about a significant release (meaningful means with some kind of functionality that will be available to users, and not just refactoring) that stabilizing an important release will take much more than a month.
With the right approach, the development of new functionality can occur quite quickly and the trick here is how not to break what was done before or quickly understand exactly what you broke and fix it quickly. Each time it is very long and inefficient to test all the details and nuances manually, because in the Elbe today there are already more than 400 “screens”. And the problem is not only that testers have to check all this manually - a lot of time is spent on the script “tester added a bug - the developer fixed the bug - the tester checked and closed / rediscovered the bug”. One can argue for a long time about how difficult it is to keep a fast-growing system in a stable state, why the code turns into a shit, why it's scary to make corrections, and no one takes the liberty to tell the release date even to the nearest month (because it would be irresponsible), but It is much more interesting to learn how to make it good.
')
I wrote the word "tests" 3 times, not because repeating words three times is fun, but because we use 3 types of tests:
Tests are written in order to run them, so it is very important that tests run easily (with one button!), Start instantly and work out quickly. Then the developers will actually run them. Our unit tests are exactly the way they behave, 1700 tests work out in 2 minutes. With functional tests, things are not so simple, because each test involves loading the page (often several) with subsequent verification. It takes from a few to tens of seconds. But taking into account the fact that we have more than 2000 functional tests in total, it takes about 7 hours to run all the tests on one machine. Obviously, there has not yet been born such a developer who has enough patience to wait for all the functional tests to pass. Therefore, the developer runs either the tests just written, or those that test the modified pages. After that, the developer makes a commit, and a build server comes into play, which, communicating with the version control system, realizes that something new has appeared and adds the commit to the queue. The queue is raked by agents — physical or virtual machines, the purpose of which is to run the entire set of tests for each commit. We use 20 agents in order to verify the correctness of work in all browsers relevant to us. You can easily see the list of fallen tests and start raising them. At the same time, we have several branches in the version control system (stable and branches in which we make new features). And for every major branch, we let in agents.
Special mention should be given to the time when we raise the tests. It is impossible to delay here. I heard that in some teams at Microsoft, all work stops and does not resume as long as there is at least one dropped test. To be honest, we didn’t try and do tests every day (the engineer on duty does it), and before each release, during stabilization (in severe cases, the whole team).
As a result, the test code in the lines is approximately equal to the system code. We managed to build a serious infrastructure for launch. What motivates us to do this? We have long understood that without tests for long periods of time, we will spend more time on development. Many of you probably more than once participated in discussions of what the next big commit could break, and then night after fire corrected what they didn’t think about and what was not found by the testers. Or you came to the conclusion that important refactoring, which will speed up the work or make the code better, you cannot afford at all, because identifying the problems that have arisen will take unpredictable time. It is very pleasant to feel control over what is happening and to understand that any change in the system is real. Writing complex systems is not an easy task, and writing systems that always work well, often changing, but resistant to change, are an order of magnitude more difficult.
We write almost all the code in pairs and strive for general ownership of the code. That is, we have no separation: server programmer, client or, say, a database developer. Any team member can take on any task. It is clear that the competence and involvement in the project or in some of its areas is different for everyone, so we use pairs to spread knowledge. And even together much more fun. Fair. And the most important thing is that it is not slower at all - when working in a pair, developers are not at all distracted by outside affairs.
In a situation of common ownership of the code, it is quite reasonable to show that the couple is going to commit to another member of the team most immersed in this topic. First, explaining to the reviewer what they have just written, the couple eventually understands this completely. Secondly, the reviewer very quickly sees some very stupid problems and points to them. Thirdly, the information is distributed from a couple to another developer - this is also great.
This, of course, cannot be called engineering practice in its pure form, just planning is such an important part of the life of the development team that it is absolutely impossible not to talk about it. Everything I wrote about above was related to quality, but quality cannot be the only goal. Good code, lots of tests and releases that are very stable, but occur every six months or less, will quickly lead the project to a dead end. This can work in a custom development, if you have a client who understands what he wants, is ready to pay for it well and is waiting for just a qualitative realization of his desires. In all other cases, more frequent releases are needed in order to be able to verify the correctness of their ideas and correct them. In the end, you need to understand what the team will be able to do in the next few weeks with a high degree of confidence. A pair of programming and general knowledge of the code requires that all developers understand the tasks in the same way before starting work. We provide this with the help of planning poker - a well-known practice of teams using flexible development methodologies. The classic approach is that the team recruits several tasks in the next iteration of development (we use two-week iterations) and votes how many “ideal” hours each task will take. Our know-how consists in an innovative form of presenting voting results, we use the following mindmap:
The letters are the name of the task, and the numbers are the hours that resulted from the game of poker. But the great thing is that we overestimate the situation every day. For each task is determined by the progress - how much has already been done. As a result, at a very early stage, we can respond to any force majeure, so we don’t need to wait until the end of the iteration to understand that the team does not have time, or, on the contrary, is ready to take on additional tasks in the iteration.
There are no bugs only where the developer’s foot has not gone. And if we consider that we have the desire to make a meaningful release once a month, that is, there is every chance to drown in bugs, because there is no time to fix them. Still it is necessary to take into account that usually fixing bugs is a rather unplanned occupation. When the number of bugs reaches a critical mass, you have to feverishly fix them, which can destroy all plans.
We face this problem and found a way out that seems to be working - the days of bugs. At first we had one day of bugs in the iteration, but the number of bugs continued to grow anyway. Therefore, we have introduced another day of bugs, which is held in the second week of the iteration. Thus, we always know that we have two days in the iteration set aside for bugs, and we take this into account when planning. It should be noted that this situation arose not immediately, but relatively recently. At first, we somehow managed without it.
WhatRussian programmer does not like refactoring? Despite the fact that the presence of tests makes it possible to fairly easily perform any refactoring, it is still not always possible to devote enough time to it for the same reason that the number of bugs grows - release once a month. If a bug can be closed (or not closed), without losing anything (and not even demotivating), then getting involved in refactoring, it is sometimes impossible and no place to retreat. To solve this problem, another day in the iteration is allocated under the "day of free creativity", in which you can do anything: refactor, do some magical things. And someone on this day, paradoxically as it may seem, has fixed bugs at will!
It is not always possible to postpone the correction of a bug until the day specially allotted for corrections. And technical support, call center and analysts every day there are various questions and requests to the developers. In order not to distract the whole team, at each iteration we choose a victim - an engineer on duty. The meaning of its existence is to answer all the questions and fulfill all the requests of the Elbe-people who are not developers.
We have a somewhat unconventional view of this role. For each release we assign a release engineer from the developers. Its goal is to ensure that all functional tests pass, the patches are prepared and tested, and no one has any questions about the release at all. This does not mean that the release engineer himself raises all the tests, as long as he raises the right questions and motivated everyone to get answers. And of course, he is necessarily present at the release, and is ready to solve any problem that has arisen.
In fact, as a rule, the duty engineer and release engineer are one person.
Acting in this way, we have managed for almost 2 years to make 1 significant release per month. The system is developing very dynamically, but we are not afraid to make several updates a week (not everything can be planned: the legislation is changing too rapidly and the growth of users in the last six months has been very significant), our engineering practices make the development very manageable and predictable.
Technical details
IDE: Microsoft VisualStudio + JetBrains ReSharper
Continuous Integration Server: JetBrains TeamCity
Bagtracker: JetBrains YouTrack
Functional tests: Selenium
VCS: Mercurial
Web server: Microsoft IIS + nginx
DB: MS SQL Server
PS Maybe it is not obvious from the text, but we have a tester. He is busy all the time and works very well :)
PPS Thank you warmly our beloved JetBrines for great tools :)
The reality is that Agile without the correct engineering practices will end very quickly. If you do not make efforts that guarantee a high level of development quality and the state of the system as a whole, then as the project becomes more complex, control will be quickly lost. As a result, you will not be able to do everything planned in the iteration, and you will only dream about a significant release (meaningful means with some kind of functionality that will be available to users, and not just refactoring) that stabilizing an important release will take much more than a month.
With the right approach, the development of new functionality can occur quite quickly and the trick here is how not to break what was done before or quickly understand exactly what you broke and fix it quickly. Each time it is very long and inefficient to test all the details and nuances manually, because in the Elbe today there are already more than 400 “screens”. And the problem is not only that testers have to check all this manually - a lot of time is spent on the script “tester added a bug - the developer fixed the bug - the tester checked and closed / rediscovered the bug”. One can argue for a long time about how difficult it is to keep a fast-growing system in a stable state, why the code turns into a shit, why it's scary to make corrections, and no one takes the liberty to tell the release date even to the nearest month (because it would be irresponsible), but It is much more interesting to learn how to make it good.
')
Tests, tests, tests
I wrote the word "tests" 3 times, not because repeating words three times is fun, but because we use 3 types of tests:
- Modular tests of business logic. Elbe is a set of several services interacting with each other (web, data storage, reference service, print service, etc.) All these services and the server part of the web are written in TDD style. About TDD said and written a lot. I believe that the main advantage of this methodology is the creation of a situation in which writing tests is not a routine or boring job. Gradually build up the tests and fanatically write the code just enough for the current set of unit tests to pass, but not a single line more is a lot of fun, it's such a mental game, like chess. In addition, honestly covering each line of your code with unit tests, you get a more perfect code. Fanaticism is appropriate here: if a code has arisen that can be erased and not a single test has fallen, this code should be erased immediately, for educational purposes. It is worth saying that the idea of ​​a good code is different for everyone: students consider the code to be compiled perfect, and some developers any code that works. Fortunately, one can not argue about this. The finest man Uncle Bob long ago summarized everything that should be summarized in the field of the formal characteristics of good code and wrote a series of articles about it. We in Elba consider the code that satisfies the principles of SOLID to be good. If you still do not know that this is not a Haskell developer, then there is a chance that you are living in the wrong way or in vain :) The code covered with modular tests is easy to build and modify: tests guarantee the immutability of the old business logic, and SOLIDITY allows you to make important changes purely locally (you do not have copy-paste and 3 classes that do the same thing?) - you achieve the stated goal, the development speed is not reduced and everything happens with the specified quality.
- Unit Tests for client javascript. Server-side business logic is fine, but end users see the web application. And they do not want to just watch the dull pages, everything should fly and explode. Therefore, we write a lot of client logic. There is nothing new in comparison with server-side business logic: the code should be SOLID and covered with tests, just another language. In this way, we test general-purpose classes and controls (for example, client validators).
- Functional tests of the web application. All unit tests can pass, but this does not guarantee that the user does not have any problems with a particular page. We do not test with unit tests for filling in the fields of each form and processing these fields on the server. To fix the correct behavior of the end pages, we use functional tests. In fact, we repeat the user's behavior: we click on the link, we expect the page to load with a certain set of fields, fill these fields, check the validation of validation, click on the “save” button and make sure that exactly what was written in the fields is settled in our repository. Even if on the page, depending on some conditions, some small inscription changes - there is a test for it. If the developer says that it is impossible to test all the conditions for which he wrote the code, then he wrote this code drunk or lying (we are talking about the usual business logic, and not about scientific applications , for example). This is how a test can look like:
[Test] public void ChangeEmailWithGoodPassword() { const string newEmail = "hello_habr@gmail.com"; Prepare(); emailSettingsPage.Email.TypeValueAndWait(newEmail); emailSettingsPage.Password.TypeValueAndWait("superpassword"); emailSettingsPage.SaveButton.Click(); emailSettingsPage.WaitSaveSucceded(); emailSettingsPage = emailSettingsPage.Refresh(); emailSettingsPage.Email.WaitValue(newEmail); }
But what happens when you run the test:
To write such clear tests, we build abstractions for them over pages and controls, thanks to which the complexity of their support is greatly reduced, the readability is increased, and even the ability to write TDD functional tests appears!
Running tests
Tests are written in order to run them, so it is very important that tests run easily (with one button!), Start instantly and work out quickly. Then the developers will actually run them. Our unit tests are exactly the way they behave, 1700 tests work out in 2 minutes. With functional tests, things are not so simple, because each test involves loading the page (often several) with subsequent verification. It takes from a few to tens of seconds. But taking into account the fact that we have more than 2000 functional tests in total, it takes about 7 hours to run all the tests on one machine. Obviously, there has not yet been born such a developer who has enough patience to wait for all the functional tests to pass. Therefore, the developer runs either the tests just written, or those that test the modified pages. After that, the developer makes a commit, and a build server comes into play, which, communicating with the version control system, realizes that something new has appeared and adds the commit to the queue. The queue is raked by agents — physical or virtual machines, the purpose of which is to run the entire set of tests for each commit. We use 20 agents in order to verify the correctness of work in all browsers relevant to us. You can easily see the list of fallen tests and start raising them. At the same time, we have several branches in the version control system (stable and branches in which we make new features). And for every major branch, we let in agents.
Special mention should be given to the time when we raise the tests. It is impossible to delay here. I heard that in some teams at Microsoft, all work stops and does not resume as long as there is at least one dropped test. To be honest, we didn’t try and do tests every day (the engineer on duty does it), and before each release, during stabilization (in severe cases, the whole team).
As a result, the test code in the lines is approximately equal to the system code. We managed to build a serious infrastructure for launch. What motivates us to do this? We have long understood that without tests for long periods of time, we will spend more time on development. Many of you probably more than once participated in discussions of what the next big commit could break, and then night after fire corrected what they didn’t think about and what was not found by the testers. Or you came to the conclusion that important refactoring, which will speed up the work or make the code better, you cannot afford at all, because identifying the problems that have arisen will take unpredictable time. It is very pleasant to feel control over what is happening and to understand that any change in the system is real. Writing complex systems is not an easy task, and writing systems that always work well, often changing, but resistant to change, are an order of magnitude more difficult.
Pair programming
We write almost all the code in pairs and strive for general ownership of the code. That is, we have no separation: server programmer, client or, say, a database developer. Any team member can take on any task. It is clear that the competence and involvement in the project or in some of its areas is different for everyone, so we use pairs to spread knowledge. And even together much more fun. Fair. And the most important thing is that it is not slower at all - when working in a pair, developers are not at all distracted by outside affairs.
Pre-commit code review
In a situation of common ownership of the code, it is quite reasonable to show that the couple is going to commit to another member of the team most immersed in this topic. First, explaining to the reviewer what they have just written, the couple eventually understands this completely. Secondly, the reviewer very quickly sees some very stupid problems and points to them. Thirdly, the information is distributed from a couple to another developer - this is also great.
Planning
This, of course, cannot be called engineering practice in its pure form, just planning is such an important part of the life of the development team that it is absolutely impossible not to talk about it. Everything I wrote about above was related to quality, but quality cannot be the only goal. Good code, lots of tests and releases that are very stable, but occur every six months or less, will quickly lead the project to a dead end. This can work in a custom development, if you have a client who understands what he wants, is ready to pay for it well and is waiting for just a qualitative realization of his desires. In all other cases, more frequent releases are needed in order to be able to verify the correctness of their ideas and correct them. In the end, you need to understand what the team will be able to do in the next few weeks with a high degree of confidence. A pair of programming and general knowledge of the code requires that all developers understand the tasks in the same way before starting work. We provide this with the help of planning poker - a well-known practice of teams using flexible development methodologies. The classic approach is that the team recruits several tasks in the next iteration of development (we use two-week iterations) and votes how many “ideal” hours each task will take. Our know-how consists in an innovative form of presenting voting results, we use the following mindmap:
The letters are the name of the task, and the numbers are the hours that resulted from the game of poker. But the great thing is that we overestimate the situation every day. For each task is determined by the progress - how much has already been done. As a result, at a very early stage, we can respond to any force majeure, so we don’t need to wait until the end of the iteration to understand that the team does not have time, or, on the contrary, is ready to take on additional tasks in the iteration.
Bugs
There are no bugs only where the developer’s foot has not gone. And if we consider that we have the desire to make a meaningful release once a month, that is, there is every chance to drown in bugs, because there is no time to fix them. Still it is necessary to take into account that usually fixing bugs is a rather unplanned occupation. When the number of bugs reaches a critical mass, you have to feverishly fix them, which can destroy all plans.
We face this problem and found a way out that seems to be working - the days of bugs. At first we had one day of bugs in the iteration, but the number of bugs continued to grow anyway. Therefore, we have introduced another day of bugs, which is held in the second week of the iteration. Thus, we always know that we have two days in the iteration set aside for bugs, and we take this into account when planning. It should be noted that this situation arose not immediately, but relatively recently. At first, we somehow managed without it.
Refactoring
What
Duty engineer
It is not always possible to postpone the correction of a bug until the day specially allotted for corrections. And technical support, call center and analysts every day there are various questions and requests to the developers. In order not to distract the whole team, at each iteration we choose a victim - an engineer on duty. The meaning of its existence is to answer all the questions and fulfill all the requests of the Elbe-people who are not developers.
Release Engineer
We have a somewhat unconventional view of this role. For each release we assign a release engineer from the developers. Its goal is to ensure that all functional tests pass, the patches are prepared and tested, and no one has any questions about the release at all. This does not mean that the release engineer himself raises all the tests, as long as he raises the right questions and motivated everyone to get answers. And of course, he is necessarily present at the release, and is ready to solve any problem that has arisen.
In fact, as a rule, the duty engineer and release engineer are one person.
Result
Acting in this way, we have managed for almost 2 years to make 1 significant release per month. The system is developing very dynamically, but we are not afraid to make several updates a week (not everything can be planned: the legislation is changing too rapidly and the growth of users in the last six months has been very significant), our engineering practices make the development very manageable and predictable.
Technical details
IDE: Microsoft VisualStudio + JetBrains ReSharper
Continuous Integration Server: JetBrains TeamCity
Bagtracker: JetBrains YouTrack
Functional tests: Selenium
VCS: Mercurial
Web server: Microsoft IIS + nginx
DB: MS SQL Server
PS Maybe it is not obvious from the text, but we have a tester. He is busy all the time and works very well :)
PPS Thank you warmly our beloved JetBrines for great tools :)
Source: https://habr.com/ru/post/128427/
All Articles