Building processes from scratch: from chaos to order

In this article, I will talk about building workflows in a small web development department. The department was formed from scratch and immediately began autonomous work, so we had to independently build production processes, step on all kinds of rakes and draw the necessary conclusions from this. In the hope that this will help someone save time, money and nerves, I will describe the problems we have encountered, and our solutions.

It is important to note that this is not some kind of universal methodology that can be implemented in any development team, and everything will be fine at once. This is just a mix of well-known techniques and practical experience, which we were able to effectively tailor to our design features. In this question there is no single simple solution. You should always build on the size and experience of the team itself, the specifics of the business, the specifics of the projects, etc.
')
The initial data of our department: a small (5–10 people), partially distributed (some employees work remotely, some in the office) a product team with customers within the company. Web projects. There are no system administration specialists within the department, but there are departments involved in this in the company.

Team communication

Let's start with building effective communication within the department. In any team, it is important to build effective channels and ways of communication, but when the team becomes distributed, this need increases. We have reached the maximum sharpness because the team is only partially distributed. We had to learn how to merge the worlds of office and remote employees.

The most important problem we faced is oral discussion. Office employees are always tempted to quickly gather over a cup of coffee and discuss the state of affairs on a project. However, in this case, remote workers will not be able to participate in this discussion, therefore, they will not be able to contribute their ideas, and in general they will not know that something is happening. As a rule, this turns into a desynchronization of team actions, repeated discussions due to the emergence of new thoughts and suggestions from the distributed part of the team and just an unpleasant aftertaste.

It became clear that some significant common things should be discussed in writing in general chat rooms or on some group calls.

This raises another very common problem that appears in teams where you need to write a lot - the problem of the culture of writing. It is no longer possible just to approach a colleague, pull the sleeve, draw something on a napkin and add gestures to your story. On the one hand, it complicates the work, and on the other hand, people develop the skill of writing their thoughts in a clear, structured form. As a result of this approach, the documentation became better structured, the tasks became clearer, each began to think about how to present his thought so that it would be understandable to others from the first reading.

All this does not mean that we have completely abandoned personal communication, voice and video calls. However, we made it a rule that the results of such discussions should be recorded in writing, as a kind of knowledge artifact that is always available to everyone.
Briefly look at the banal list of tools with which we solved our communication tasks within the team.

To start, we started a chat in Telegram. But then, due to the growth of the team, we realized that in one chat we were already crowded, and switched to Slack. There we broke the general flow into thematic channels and established clear rules for which reasons to write to which channel. This helped to avoid mixing of useful information and flooding.

Plus, the transition to Slack gave us some nice features for automation and integration with other services, such as repository management system or bugtracker.

• Audio and video calls - Skype for Business.
• We conduct tasks in Jira.
• The knowledge base is stored in Confluence.

Planning, execution and control of tasks

When we built communications, the problem of setting, controlling and performing tasks within the team became relevant (I will talk about working with the customer in this regard a little later).
We did not have a single list of tasks, their priorities, readiness status, etc. Instead of a clear, transparent and traceable plan, there was spontaneous short-term planning and execution.

To deal with this situation, we began to use bugtracker (in fact, we tried five of them). The outlines of the general direction of the project began to appear, an understanding of the state in which particular tasks and the overall picture appeared. However, we faced the problem of lack of discipline when using the bugtracker, which began to depreciate many of our undertakings. Not all tasks were started in a bugtracker, existing ones were not always actualized, etc. Simply put, the picture of the state of the project has ceased to be relevant and reliable.

To combat this, we have developed and implemented our own task management culture:

• Work should not be performed until the appropriate task has been initiated. It helps to always keep up to date the development history of the project and the work of the team.
• When working with a task, it is necessary to change its statuses in a timely manner. In our case, there are four statuses missing: “To do” is the starting state of the task, “In progress” is a task that is in progress, “On hold” is a task that has been started, but the work has been suspended (we are waiting for some additional information ), “Done” - the task is ready. The readiness of the task must somehow be confirmed by the customer or manager within the team.
• Over time, the number of tasks and projects has greatly increased, so we have divided the general list of works into separate subprojects, and the task should be started according to this list of subprojects.
• The task must be prioritized. This helps to understand which tasks in which order you need to perform in order to cause the project maximum benefit.
• Periodically conducted review of tasks and their priorities. As projects live and develop, and business plans sometimes change, over time some tasks become less or more relevant or even need to be removed.
• Some key discussions on tasks that were made orally or in writing in a chat require fixing the final decision in the task itself so that when it is carried out we always see the most relevant information about it and about its history. It often happens that the initial formulation of the problem after a series of discussions is transformed into something completely different, and we need to follow this.
• If, within a team, a task is transferred from one group of specialists to another, then the transmitting group is obliged to fix in it all the necessary knowledge artifacts that need to be transferred to the next group. For example, a design group must attach mockups and all the documentation necessary for development before transferring the task to the development team. This avoids unnecessary questions, distractions and context switches.
• Attaching commits to tasks. With the help of some commit naming rules, we automatically link to tasks with links to commits in GitLab. This greatly helps in the development of understanding who, what, how and when did this task. And in the opposite direction, by correctly named commits, you can always find a task containing the reason for the changes made.

Communication with the customer

The next category of difficulties that we needed to resolve was working with customers. The first thing we had to eradicate was the setting of tasks in words. If we have implemented a culture of writing within the department, then it is time to extend it to external contacts.

It’s no secret that another manager likes to approach the developer, verbally tell him what to do, to persuade to poke his fingers on the screen and leave, hoping that everything will be done well and on time. In this situation, the manager and the developer can be replaced by the product owner and a certain manager of the development team who conducts all these tasks. The result of this will not change.

In this approach lies several problems:

• What is said in words and gestures will be forgotten (at least partially) approximately tomorrow, at best, the day after tomorrow. It is not even so much about some trifles, but about the danger of completely forgetting about the task. At the same time, it is impossible to confirm in any way that you put it to someone or that someone promised you to do at least something.
• Usually, such a statement of the problem is rather messy, and there are no deep details in it. As a result, it is very difficult to clarify the missing information.
• As discussed earlier, the task statement in words indulges the pernicious unwillingness to learn to formulate their thoughts in writing so that other people understand.

When you have many customers, and each has its own characteristics, it is difficult to build a unified procedure for working with everyone. Ideally, we would like to lead all customers to a bugtracker, where they could set tasks, set priorities, discuss details, take work. But it still remains an impossible task. However, we were able to find a compromise in which tasks flow to us in the corporate mail of the department in a strictly written manner, and from our side the manager starts and leads them further in the bug tracker according to the rules already established by us. Tasks have ceased to be lost, forgotten, stored in the heads of specific people, discussed with an incomplete composition of the required specialists, etc.

Next, we faced another important problem: the customer is difficult to formulate the problem. The customer is not always competent enough (or rather, practically always insufficiently competent) in the formulation of the technical task for the technical team. And that's fine. The human factor cannot be ignored either: it can be trite and awkward for a customer to come asking the team when he himself has not yet been able to formulate it to the end. One of the criteria of a mature professional team is the ability to help the customer in identifying his problems, requirements and solutions.

It often happens that the customer, instead of coming up with a problem, comes in with a request to implement the solution he has already invented. In order not to surprise either ourselves or the customer with the results of the work on the TK compiled “on a napkin”, we created a basic check-list of questions for the customer. Already on the basis of these answers, it is easier for the customer to understand what he really wants, and the development team, what is required of it. And then it is the turn to ask some leading questions to clarify or identify the requirements.

So, before the first meeting with the customer, we ask you to first fill out (as far as possible) and send us this check-list, so that you don’t have to waste time on the same type of questions, but immediately start a fruitful dialogue. It should be noted that when interacting with the customer, it is important not only to clarify the answers that he provided, but also on the basis of the stated problem to help him identify those requirements that he could not think about.

List of questions for the customer:

• What is the purpose of the project? What problem does it solve? What business value does it carry?
• Is this the only possible solution to this problem? If not, what other options are there?
• Are there any general requirements that go through the whole project? For example, if it is an online store, then it must fully comply with the legislation governing online trading.
• Are there any functional requirements? What should the section do (page, project)? For example, the section should provide information about the company's products and collect those who wish to ask questions about this product or purchase it through the form on the page.
• Are there non-functional requirements? How well should he do it? For example, the page opening time should be no more than 5 seconds.
• Additional requirements. Free format in which you can pour out the soul.

Another problem that we had to face: tasks coming simultaneously from all sides. When there are many customers for different projects, everyone wants to set the highest priority for their task. In the general ideal case, probably, such a problem cannot be solved 100%. How do we live with this? With the introduction of discipline in the formulation and maintenance of tasks, as well as some elements of the Agile-methodologies, our task pool has become more streamlined, transparent to an external observer, and most importantly - predictable. We have established order and plans within our team, and it remains for us to only strengthen the feedback with customers. In the discussion of priorities, deadlines and plans, we learned to build a reasoned dialogue, and not blindly rush to the tasks and constantly extinguish the fires that have flared up (which are actually not always relevant and not always fires).

Also under this item, we tried to eradicate the classic anti-pattern “tea-management”, when the customer or manager flew in, “imposed” the tasks - and flew away in full confidence that once he set the task, he would certainly get an excellent result. As practice has shown, the result with this approach was not very impressive. How to deal with it? There is also no universal council, some kind of magic phrase that changes people's behavior. For this you need to talk, educate, explain, show, you can say bring up. Only educational work and either very illustrative or very measurable (and preferably both) positive and negative examples will help to sufficiently overcome the “tea-management”.

Dev vs ops

And our last important problem is a problem in communications between the Dev and Ops departments.
We are faced with a classic situation where developers are not very well aware of how the server works, and the third-party admin team does not really know how the application works. Every difficulty at the junction of these two spheres was given to us with pain and time consuming. It was even difficult to diagnose which side the problem was:

• That you programmed there!
• No, you have something with the server there!

In this case, we were helped by the fact that a developer appeared in the team who was well versed in administration. He was able to speak with each group in its language and diagnose every problem on both sides. The inconsistencies went into decline, but we remained tied to this man. In order to untie all these problems from himself, he began to help administrators understand how the application works, and programmers what is happening on the server. Let us add documentation documentation to the voice explanation and get not only knowledge of the whole team, but also more coordinated work of the Dev and Ops departments. Yes, in large departments of a bloody enterprise, there are special teams and qualified people who will set everything up so that developers may not even know how and where their creativity works on the server. However, in small teams it would be nice to develop this level of competence in people, and also to have at least one employee who is already quite well developed in this regard.

Development

In parallel with all these, we took up the development of a culture of development.
I will not focus on the technical part, and even now this is already a de facto standard and almost everyone has an understanding of how a version control system, CI / CD and other development tools, assembly and deployment tools are needed. I will dwell only on the soft moments of development.

• Codestyle . It is quite important to work out and explicitly approve the code design rules. First of all, it is aesthetically more pleasant to see in the project a single slim appearance, and not a zoo of different styles and standards. Secondly, it increases the readability of the code, and as we know, most of the time the programmer does not write his code, but reads someone else’s.
• Naming commits . We have introduced certain rules for naming commits, so that by the name of a commit it is clear what changes were made, by whom, and within the framework of which task.
• Code Review . The specificity of our projects and teams is such that we don’t have any obligatory code review, without which we don’t have to infuse our code into production. However, we made it a rule that at least one person is watching the code that a colleague is pushing. It helps both to notice any shortcomings, introduce alternative ideas, and just to be aware of all parts of the system. The code review has become particularly relevant with the growth of the number and complexity of projects, because of which each developer no longer has time to work on all projects enough to understand all their details.
• Coordination of the architecture within the team in the early stages . It often happens that in an effort to create a feature quickly, the frontend starts to do something of its own, the backend quickly starts its own, and then it turns out that it integrates with great difficulty or does not integrate at all. In the development of large features, we jointly discuss in advance the architecture, data exchange formats, etc. As a result, integration has ceased to be long and painful.
• CV Driven Development . This is a problem in which developers are drafting new (fresh, trendy, high-paying) technologies into the project not for the sake of the project, but for the sake of a tick in the summary. We are open to new technologies and are trying to develop our projects technologically, however, it is important to maintain a balance here, with technological progress and projects successfully completed within a reasonable time. In this slippery topic, two things are important: so that the customer does not press deadlines in such a way that developers do not breathe, and that the development team (or, at least, a competent team leader or technical leader) was concerned not only with the development of his LinkedIn profile, but also with the project’s well-being. generally.

Refactoring, technical debt and the principle of continuous improvement

Our department began to form around an existing park of projects made by outsourcers. Naturally, there was no documentation, no one followed the relevance and quality of the code. Since we understood that we would have to work with this for a long time, maintain and develop it, it was decided to allocate part of the time for restoring order in the project — refactoring, deleting irrelevant code, etc.

Since the project was large, the level of entropy in it was also high. We realized that in one sitting it would not be possible to physically overcome everything, and the hands of the prospect of such a colossal work are morally lowered. We decided to apply the Japanese principle of continuous improvement kaizen. We broke up technical debt into many small parts and gradually, but regularly closed these small parts, continuously modifying and improving both the projects and the work of the team itself. Morally it did not bring any inconvenience, but it did not have a significant impact on the development of new functionality and coverage of business requirements. After a year and a half, we found that the old technical debt was fully repaid, and this opened up opportunities for us to develop a functionality of a fundamentally new level of complexity and importance.

Of course, this does not mean that we now have no technical debt. As projects live, evolve, grow and develop, it certainly accumulates. However, we carefully monitor him, postpone it in a special task pool and regularly allocate some time to pay it off. Such continuous work with technical debt allows us to find a balance between the development of new functionality and the quality support of the old.

In our case, we, the developers, managed to explain and show the business the importance of repaying technical debt. How did we do it? In practice, we have demonstrated such situations in which, if we do not refactor or some other structural changes in the current project, the development of new functionality or the change of the old one will be impossible in principle (or possible, but several times slower).

Implementing Agile Methodologies

The introduction of some ideas of Agile-methodologies allowed us to increase the transparency of our work both within the team and for business, to make the development more predictable and flexible, and the result is more stable.

The first thing we did was organize daily stand-ups within the team. Since the team is distributed, in Slack they took a separate channel for this, in which every morning each employee writes three points: what tasks he worked on yesterday, what plans to work on today, whether there are any problems hindering his work. Flooding in this channel, discussing someone's problems or problems is prohibited. This channel is strictly for the aggregation of status information. The remaining discussions should be conducted in the appropriate thematic channels. This allowed each person in the team to understand what his colleagues are doing, what is happening with the project in general, who can and should be helped. If, without stand-ups, problems were hushed up, and after a long time, it suddenly turned out that the task was not ready yet, it is now clearly visible who needs help, what needs to be done to make the team work more efficiently.

Next, we decided to lead the development of sprints. Every Friday at the end of the day we do a sprint retrospective, look at what tasks were in the plans, what is ready, what is not ready, and if something is not ready, then why it happened. We think about what problems we had and what we could do to avoid similar difficulties in the future. Then we plan a sprint for the next week based on the workload of different directions within the team and business priorities. As a result, at the beginning of the week, all developers know what to do with them and in what sequence, and the business knows what its needs will be covered in the near future, and can already form their wishes for the next sprints. It is worth saying that we are not spared from sudden tasks that may disrupt our plans for the sprint. In this case, you need to act on the basis of the specific specifics of the work: how often do such tasks arise? How much? Can this be predicted? In our particular case in development, we empirically calculated how much time, on average, we have for unplanned tasks and try to lay this stock in the sprint. Separately, it is worth noting that after we started working on sprints, we were able to specifically find out how much work we are delivered to the input unscheduled, analyze and reduce this number, carefully discussing priorities with the customer and clearly showing how the momentary desire to get the least necessary feature right now on the overall productivity of the whole team.

We also moved from long to short releases. Previously, we received TK, for weeks or months we made a whole pack of features and only then showed it to the customer. As a result, it often turned out that the customer either changed his mind or didn’t expect it at all, and we began to redo all or part of what we’ve already done. And to make changes to a ready-made large set of features is a dubious pleasure. Now we are demonstrating every feature as early as possible so that the customer can decide whether it is what he really wanted or whether something needs to be changed. The sooner he either approves or sends for revision, the less labor we invest, therefore, the faster the feature will come into production. As a result, features began to arrive in production faster, hypotheses were checked faster, the development of the project went faster.

Bus factor

Since our team is small, we immediately began to think about the problems that we may encounter during the rotation of personnel. Specific people have become the sole custodians of some knowledge, projects have grown enough in breadth, and so we began to develop a culture of knowledge storage and management.

The accumulation of knowledge artifacts that have moved from the heads of specific people to the physical written world helped us eradicate this problem. Namely:

• All work is carried out only if there is a task in the bugtracker. This allows you to make a complete history of changes in the project.
• If somewhere in the chat or at the rally, we discussed the changes on the task - we will necessarily reflect the result of this discussion in the task itself.
• The commits are named with a clear indication of the changes made and a link to the task. At the same time, integration in the bugtracker is configured so that in the task itself there is a link to the commit in Gitlab. This allows you to save the history of not only the goal of the task and its discussions, but also save information about specific changes and their performers.
• After developing a new feature, the programmer writes documentation to Confluence, describing its general business logic and implementation features, if necessary.
• If any major problem incidents occur, the postmortem is kept in the format “what happened when - what they did - what they really had to do - how to avoid such a problem in the future”.

There is still a good practice in which, of course, you need to know the measure and not to build it into an absolute: if you are asked about some part of the system, about which only you know, then it is advisable to write documentation about it and respond with a link to it. So you will never have to answer this question again, and ask other people it.

This method of maintaining knowledge artifacts has repeatedly helped us find information about those parts of the project that would otherwise have been lost. He also significantly reduced the risks for the project and the team during the rotation of personnel. The last example: we quite easily and quickly recovered information about the business logic, the principle of operation and the way the tool was deployed, which was done by the employee who left two years ago.

If we impose work with stand-ups and sprints on this technique, the bus factor is reduced even more. Not so long ago, we conducted an experiment: one developer was on vacation, and another developer was working. At that time, when the first came out of vacation, the second went on vacation. The essence of the experiment was not to write any explanatory letters, messages, not to transfer cases and see how difficult it would be for a person after a long vacation to understand what was happening in his absence, what had changed, how exactly and what plans for the future. As practice has shown, viewing the bugtracker, commits, documentation, stand-ups and sprints allowed the employee to easily get up to speed and autonomously continue his work.

Conclusion

In conclusion, I would like to note that none of the above problems were solved immediately and flawlessly. Organizational change is always a long and methodical work. You can’t just say “now we do it this way” and hope that now everything will be different. Any decisions that you make, any activities that you organize, require monitoring, training and education of people, time to adapt as a team to new methods, and the methods themselves to a specific situation. I also note that imposing a methodology on people is a very laborious and inefficient process. People will come up against, sabotage (s / i), maybe even sabotage.

In order for the team to begin the required changes, the team must itself want to make these changes. It is necessary to monitor how things are going with her, identify problem areas, be aware of these problems, find solutions, and only then implement them. Of course, not every team member should and wants to do this, but if there is someone who has seen these problems and has come up with their solutions, then he will first of all educate the team.

Share your knowledge, observations, experience, argue, show and prove where the problems are now and what steps can be taken to solve them. This is the only way to make the most effective organizational changes as effectively as possible. Even if you are a manager and want to push your position and your decision - try to do it as convincingly and convincingly as possible, thus you will save time on implementation and save the team from unwanted negative.

Author : Evgeny Antonov, Head of the Positive Technologies Development Team