What to do with other people's debts?

One aspect of the developer's profession is the dedication of the profane in particular the software development process.
S. McConnell, Perfect Code

The purpose of this publication is to share the experience of working on a project with a complex history and a difficult legacy. After leaving the next so-called. “Startup”, I decided that I want to try new sensations: enterprise, legacy, etc. To do this, took up the work on a corporate application for a transnational concern. Development at that time was already the third year, the application has experienced several generations of developers, but there was no stable release.

I believe the publication will be useful:

• developers making a similar decision to weigh the pros and cons
• managers of "difficult" projects to better understand the causes and consequences of technical problems
• and of course just curious

Issues covered in the article:
')

• Low developer competence, and what can you do about it?
• What arguments are convincing in the eyes of the customer for non-functional changes in the project?
• Why is the work of analysts and QA very important from the point of view of development in particular and for the project as a whole?

Input

What is at the entrance? At the interview I was intrigued by the following details:

• The used version of the framework over the years has become obsolete
• The release of a stable version of the product in two years did not take place
• The business complexity of the system sometimes rolls over

At the interview of the second stage, with the participation of the project manager and lead developer, I received the following batch of nuances:

• The current development team has a low level of competence in the framework, language, testing, IoC, object-oriented design, design patterns, and architecture used
• The subject area is really not easy.
• The customer is convinced that there are no problems

By the time a positive decision was made to participate in the project, I did not see any good or reliable place in it: absolutely everything that I was able to recognize was a sign of problems. That is what pushed me to a positive answer =)

Summarizing the state of affairs at the start, the following ballast categories can be distinguished, which represented a risk to the success of further development:

• Low developer competence
• Low quality code (resulting from the previous paragraph), a large technical debt
• Lack of established infrastructure
• Lack of automated testing
• Lack of acceptance testing culture
• Lack of culture of versioning, modification, deployment
• Requirements not standardized
• Manager's duty to the customer - hiding the problems on the side of the executing team, exaggerating success, new promises to old debts

The general message that I would recommend to developers applying for the role of "rescuer" (in the position of senior, lead developer, technical support, etc.): do not lie and not be afraid . If you find the project in a deplorable state, it does not mean that the people who worked in it before you consider it the same. Firstly: the current state of the project is in many ways the fruit of their efforts, it is psychologically difficult for people to perceive criticism of the result of their work. Secondly: most likely, the level of the team that admitted the existing problems does not allow them to be discerned - for them objectively there are no problems. In the described case, the banal fat controllers - containing all business logic - were considered acceptable. The absence of unit tests is not perceived as something terrible, if a person is simply not familiar with unit testing.

In this situation, you will inevitably have to impose your best practices on the team. There are two ways to do this: evolutionary - to train (first of all by your own example) the available frames. Revolutionary - to seek more competent performers, and therefore systematically replace the team.

We start with the tests

One of my first events was to connect a test framework, help my colleagues in launching and writing tests. Evolution, of course, is a more humane option - you don’t need to “merge” anyone, spend time and money on the selection of more experienced (and probably expensive) specialists. And karmically, it is much more useful to share knowledge. But this is a long way. If a developer mature enough to consider himself ready to develop enterprise applications for money, didn’t master testing, something in his career curve went wrong. The drama is that most likely, unable to meet the new requirements, many will prefer to find a new job. It is human nature to follow the path of least resistance. In this case, an attempt to train is a waste of one’s own and other people's time.

If you do not have an extra person-year left, which is more than likely for commercial development, then an inexperienced team has little chance of smoothly evolving. It is better to immediately prepare to form a new team, making demands to applicants higher than it was before. Yes, thus increasing the budget. And this is not a swelling of the budget, this is a compensation for the debt created in the past by the manager: maybe in an attempt to save money, maybe due to lack of experience.

Enumeration of technological changes that were to be made, I started with unit tests. Unit tests are at the same time the cheapest opportunity to implement and maintain to keep your project in order. And at the same time one of the most fundamental. Starting with tests is a good habit. With tests, you can start coding, using an unfamiliar library or language, setting a task, checking the implementation of something. Tests are a way of thinking in which you formally describe the result before you do something. If you hone this skill to the proper level, then getting any result will not cause you any difficulties, you will forget about the fear of the white sheet. And this concerns not only programming.

Svn -> hg -> git

Historically, the project used SVN. The Mercs were a difficult and responsible business, accompanied by a sacramental "not pushta while in trunk, today is deployed." Department colleagues in other projects used Mercurial. For about a month, we tried this version control system, but in the end we preferred to switch to the well-known and popular Git. As expected, the majority of applicants, when forming a new team, were more often familiar with Git than with two other hard currencies. What is not a reason to go?

CI & CD

Windows Server -> Ubuntu
Remote Desktop + manual update -> delpoy.sh -> Unix + Docker + TeamCity

Copies of the application for demonstrations and testing were on the Windows Server. Managing the server and updating applications was done manually by connecting to a remote desktop. About half a year it took me to convince the manager, and, through him, the customer, that the transfer of infrastructure to Unix should be a prerequisite for release into production. In parallel with this formal justification, in the process of searching for the second backend developer, I looked towards the candidates owning the LAMP stack administration. Fortunately, we managed to find a specialist with good skills in Bash and Unix: as a result, he became a team 50% developer and 50% build and integration engineer. By the time we entered production, we had a full CI and CD . Hi Rottenwood !

This event, like others, is not a purely technical solution. Methodologies and design concepts affect other processes, don't forget about it. If the manager is used to lead a team for which release preparation is reduced to “** yak - ** yak and in production!” , It is not enough to set up an agent in TeamCity. You will have to convey to the manager the realization that “you can’t just take it and fix it in five minutes before the demo” . Yes, it will be at first discomfort. It will take the manager a month or more to get used to what the deploy now does not happen on a kick, instantly, along with the fall of the working production. Now this is a deliberate procedure, which even if it takes 10 minutes, but is guaranteed not to drop anything, and will give a predictable result on any of the servers, no matter how many of them you have.

For the customer, the arguments for the need to allocate resources and time for this were:

• Reduced downtime in emergency cases - thanks to Docker, we can quickly deploy on any virtual machine in any data center.
• We can put the application together with the image and it can be run on any machine (this case was also considered) without participation from the development team.
• Security - the isolation of the container with the application from the host machine, the simplicity and reliability of Unix for the server. Part of the points when the application passed the corporate security audit was automatically closed.
• Native environment for the stack used by the application, a wider range of applicable technologies for potential features. Great developer loyalty.
• Tangible performance gains without additional configuration.

Eclipse / NetBeans / Trial WebStorm / Brackets -> PhpStorm

Another important event was the organization of the acquisition of the license for PhpStorm for the team and the setting of a uniform formatting style in accordance with the PSR. I think any organization can ( and should ) provide workers with normal tools. PhpStorm and WebStorm are now the leading, in my opinion, IDE on the market for PHP / JavaScript / TypeScript support. A good IDE can significantly improve both the personal effectiveness of the programmer and the team as a whole - it is easy to implement through the settings a single code style and various useful “gadgets” for working on the project.

Devprom + Excel + * .jpg-> Jira

This transition was probably the most epic and long-awaited for us. Historically, Devprom has been used. In a nutshell: I do not recommend this system to anyone. For me it was a revelation that paid software can be so low quality ! Randomly the system could hang, fall, contained frank vulnerabilities. Each update, in addition to the patch of several SQL injections (and adding new ones, judging by the frequency of updates), introduced innovations in the layout of GUI elements, so that the usual usage scenarios had to be re-learned.

Since the project management system is the cornerstone of the planning and development process, the use of a “buggy” and unstable solution affected all other processes: people were too lazy to get bugs in the tracker (it was difficult and time consuming), when planning the backlog of iteration turned into torture for the whole day , tasks were lost, analytics and assessment tools were easier not to use, managers tried to push something into the work around the tracker whenever possible. Thus the tool used clearly damaged our processes, agree it is terrible.

Thanks to Devprom and the inventiveness of his accustomed managers, I observed some alternative methods of management:

• Work on pictures - after testing at the exit, we have a folder with screenshots, named according to priorities. The developers have fun enough to grab the images they like and “fix” them, shifting the pictures to the Done folder. For trying to create a task for each picture in the tracker, it was possible to get a handle.
• Email-driven managment : in the mailing list between team members for several days to a week, a letter walks in the table of high-level features. The initiative to use the tracker board for the operational management of the development is punishable.
• Excel planning - backlog compilation and evaluation using the legendary tabular processor.

Task tracker is no less important in development than IDE. This is the cornerstone of the development process : any activity in a project must begin and end there. No task - no code. Jira, perhaps the best solution for commercial organizations in the market.

PM-level

If a developer exercises operational control in a team, a lot can depend on his decisions. His choice determines the success or failure of the implementation of individual parts of the product. Of course, this is most relevant for small teams: 2-4 developers, tester, analyst. In proportion to the increase in the number of different specialists - we introduce architects, administrators, QA-department - I suppose, the degree of personal responsibility of an individual participant in the process is reduced.
But do not forget that there are at least two factors that you, as a technical specialist, for whom you have no direct influence. At the same time, the very possibility of project success directly depends on them:

• The total budget of the project and, accordingly, the staffing. You can not write a competitive analogue of the application, if you have a staff of an order of magnitude less specialists. Beyond the prototype, on enthusiasm your “startup” will not fly away.
• Activities in front of you in the food chain - the customer and business analyst.

If the project manager is on your side and can knock out the higher management (or customer, depending on the structure of your organization) additional costs, if the manager is able to defend the customer with the importance of the non-functional changes in the project and process, you have chances for success. The rest depends on your perfection.

If the management is not ready to listen to the development team, there is no chance of recovery. You cannot directly influence the decision of a superior, or change the project management. But you must do everything possible to be heard, if you do not want to be responsible for the mistakes of others.

Dev-level

If your developer experience suggests that the project has problems, such as a banal technical duty, you can throw your chest to the embrasure of refactoring ... And die, since the rest of the team will have time to gossip more machine lines per minute from your machine gun than you can physically refactor and cover with tests. Abstract from the development itself, look at the process from a bird's eye view: code reviews, protected branches and pull requests, statistical code analysis tools in your CI - there are many tools to prevent the spread of the problem. It is much more important to eliminate the cause of the problem, elimination of the symptoms is a secondary matter. And not the fact that you have enough time for the second, with the majority of legacy will have to live for a long time. The main thing to prevent metastases.

Sometimes the error is really in the genes. Then the best way is to cut off the poor-quality genotype from your population and replace it with a fresh sample with the necessary dominant traits.

After a year of refactoring, it seems to me that sometimes developers should be assessed not by the large amount of benefit brought (functionality + corresponding code base), but by the minimum damage left, in the form of technical debt and unsupported or untested code. Of course, your manager will have an alternative view of reality. But the reality of development is such that it is easy to “cut down a feature” of almost any complexity for an average developer. Much more important in the medium and long term that this addition does not degrade the architecture, was accompanied by not fragile and understandable tests, was designed with an eye to the principles of SOLID. In this regard, I prefer one senior to two middle and two middle to four junior. The longer the distance that you and your product have to go, the more important this thesis is.

Alas, it is almost never possible to assemble a strong enough team in a reasonable time, even with the necessary budget. Skilled specialists even in the most widespread technologies and frameworks are sorely lacking. Building a development process at an industrial level will help you compensate for this. Use possible utilities and techniques to analyze and control code quality. Putting a well-established conveyor of an average or novice developer, you will get a more predictable result and a systematic development, than by letting it be committed to a master enthusiast with burning eyes. Your task should be to organize a stable process and facilitate embedding participants in it, rather than a heroic elimination of the consequences of no less heroic dashing handicraft decisions.

Analytics

If you see that the business analyst throws off raw requirements in the development, do not turn on the fantasy and do not start coding. Make a list of questions and all conflict points and send him a letter. Or discuss all doubtful points together over the printout. Encoding is to start when you have such specific requirements in your hand, in a file approved by the analyst, that the task of encoding can be delegated to any developer. I believe that the ideally assigned development task, besides the reference to the relevant requirements, should not contain any “business logic”, in the extreme case, a high-level description of the design patterns used, if the designated developer is not yet familiar with the components of the system where changes are expected.

The banal truth that we daily forget: people tend to think their thoughts are obvious . Despite the average high IQ in our industry, telepaths, alas, are always on vacation. If you enter the telepathic mode and do his work for the analyst, then you will find yourself guilty of the following sins:

1. With a probability close to one, the task takes you more time than you planned. After giving a rating, the developer always voices the time for coding. The most experienced of us can lay the risks of debugging, testing, documentation support, etc. But I doubt that even the best of developers are able to accurately predict the time they need to work as a business analyst. The analyst will not program for you. And for the exceeded terms to answer you.
2. You will be responsible for all your fantasies, because when they get into the implementation, they remain not described in the requirements and not agreed with the customer. Then you will have to convince everyone for a long time that this is “not a bug but a feature”, and ultimately rewrite this code, with the appearance of real requirements. Well, if you are. It is much worse to stumble upon undocumented behavior and mysterious code with initials whose owner nobody remembers at the project.
3. You are depriving a good analytic guy of professional growth opportunities.
4. You increase the entropy of the universe, leveling the benefits of the division of labor.

The results of oral discussions and correspondence should be recorded in the final files with the requirements. Develop a package of documents necessary for adequate development planning and planned coding. Determine which formats are needed to describe one or another part of the system. For example - for each screen form - the following set of documents can be claimed:

• Visio file with a description of the functionality, flowchart dependencies between fields
• Excel file with validation rules and description of data types for each field
• PSD file for coder, associated sources (fonts, icons)

Such a package will be in demand not only at the development stage but also at the following: acceptance testing and development of user documentation.

It would be ideal to version these files in parallel with the source code in order to understand the current and required state of the system. Unfortunately, with most complex office formats, versioning by analogy with source codes in Git does not apply.

Business analysis is a process that is important to formalize and debug no less than the development itself. And since in a developer-analyst relationship you are a kind of consumer, it is your duty to help build these relationships for mutual pleasure.

Estimate it

If the manager forces you to evaluate the task for which the analysis was not conducted, put the top one possible. For example, I use Fibonacci values ​​of 13 for such tasks, or 21, while for normally planned tasks, the maximum value is 5. Thus, you clearly reflect the complexity, which at this stage cannot be assessed more accurately.

At the other extreme: set the minimum score. I use 1, although many optimists tend to make promises like "this can be done in 5/10/15 minutes." Yes, of course there are changes that make it take a few minutes - not counting the overhead of interaction with the tracker, hard currency, documentation, tests. In order not to upset the manager with the fact that the “small” fix takes a whole engineering hour, I can recommend the associated minor edits to be combined into one task.

QA

If you get a bug report in the form of “Fix a form on page M” or a single screenshot with a large bold question mark on a harmless, familiar page, you are unlikely to be able to fix the problem. Formalize the bug report format to fit your application. Show and tell everyone involved in testing the product how to obtain debug information needed to correct, how to formulate. Do not attempt to reproduce the unplayable.

Another caveat: if the team does not have a testing culture, the manager may believe that manual testing of the product is a matter of developers. Your mission is to show him simple arithmetic: the developer hour is usually N times as expensive as the “manual” tester hour. For several full days of testing by the developers who are available in the team, the salary of the dedicated tester is easily burned. Do not forget about the simple development.

Testing is a process that should be conducted systematically and on a regular basis, and not an event like the April work day. If your organization does not have QA yet, but you know what it is, bring it closer by all available means. While there is no qualified acceptance testing, the development team will be the last in all detected bugs. If testing is irregular in nature, then the bugs will be rarely, but a lot and not those. This means that an avalanche of emergency bugs will poison your life and seriously harm the entire development process. In the above-mentioned project, it took me about a year and a half to knock out a full-time unit for a QA specialist and search for a suitable candidate.

What are the risks of irregular and poorly organized testing:

• If testing is rare, it will seem that the version is terribly “buggy,” the blood-guts-plague, aaa, everything is lost, everything is abandoned, everything is running to the fire !! 1 - this disrupts the planned development, sometimes completely paralyzing it. Risk to disrupt the next deadline.
• The one-time detection of a large number of bugs will be subjectively perceived as something that is critically many in the (seemingly) ready-made version, which undermines your developer credibility.
• In the case of manual, episodic, poorly organized testing, bug reports will be of poor quality - and therefore difficult to reproduce, irrelevant, duplicated. Developers will spend extra effort and time trying to play them. In my experience, I can say that, often, the reproduction of a bug takes no less than a fix. And doubly sad for the time spent, if it was not possible to reproduce.
• The time to fix a large number of bugs is more difficult to estimate compared to regular daily inclusion in the plan. If the process is debugged, then with the performance metrics of testing and development, it is possible to lay the evaluation of corrections in a long-term plan, in parallel with the rest of the development.
• The chances of “fixing” all the bugs that became known the day before the release will be very much desired by the manager and the young developers on time. But no, it does not happen. The release will either be of inadequate quality or disrupted.
• Experts who perform testing on a occasional basis (visiting trainees, project managers, analysts, the developers themselves (God forbid!), A coffee lady, an accountant and plumbers) by definition: a). low-effective in this role, since it is for them occasional b). are unlikely to act agreed upon under any plan or scenario c). low quality reports guaranteed

To help organize healthy testing is in the developer’s own interest.

The best is the main enemy of the good!

If you are lucky and you debug the development to the ideal state, this does not mean that the management and the customer will automatically become happy. First, if PM, for example, chronically flexes for the customer, taking without the knowledge of the team development commitments more than physically can be done, then as the stability and productivity of the development increases, so will the commitments. Secondly, there will always be problems that were not voiced earlier, which for the non-estate of others will receive the highest priority. Here is the scheme: if before the command “fakapila” stability and new functionality, now the customer can consider that the application is slow and write it as an “epic factor”, although initially there were no requirements for this. Or recall some Pet-feature, breaking all the existing logic and put it as a must-have, and any counter-arguments to consider unprofessional and sabotage. Here everything depends on the adequacy of the customer and your managerial layer. In such a situation, it remains only to consistently argue, appeal to logic or look for a more grateful organization.

One more thing to keep in mind: as the product grows, the cost of making changes inevitably grows . Is always. All the best practices and other gadgets described above only reduce the growth rate of this value. Thus, if a person has not had a long experience of working with teams that have well-established processes and understands these processes poorly, he can draw the following conclusion:

- it was: no formalism, ** yak and on sale, the boys quickly did everything.
- became: a lot of formalism, CI / QA-some kind of interferes with a quick “release”, the boys slowly began to code ...

At the same time, without having personal experience in projects of various sizes and teams, it eludes him, that between "was" and "became" are:

1. large time period
2. a large slice of the functionality implemented during this time
3. unpaid debts from the fact that "was"

Therefore, for such characters, it is important not so much to put things in order - for a non-developer, any changes, other than functional ones, will be ephemeral and optional. It is important reasoned justification with good examples: what, how and why. The very aspect of the developer profession that McConnell mentions.