hh and in production: how to release a new feature

Once a technical department in HeadHunter made sweatshirts with a print on the back "hh and in production". It seems to be funny, but all the time I was confused by this inscription, because in the original it is not something to be proud of. This prompted me to write a post, what the letters hh actually mean.

I want to talk about how the feature is born in HeadHunter using the example of the API command in which I am working. What path she has to go from idea to output in production. I will touch on both technical and managerial aspects.

We are giving birth to children so easily and carefree, but we care so little about the creation of man! We all yearn for some beautiful man. In our will to help him appear on earth! So let’s use up our will so that he will come sooner, and maybe we will be rewarded for this happiness to see among us young forerunners of whom our soul longed for.
Maksim Gorky
')
At once I want to define that a certain business unit of the application is meant by features, something that the end user can touch.

We have 17 teams in the development department, all of them are self-organizing - this is one time, and two: each team has its own specifics, features inherent in their occupation. Therefore, in other teams workflows may differ. I will tell you how the developer of the API team sees this.

Stage one: idea

So, we need a new feature. For her to be, for a start it is worth coming up with. Different people can come up with it, but more often it is done by those who feel the need for it. We will call such people customers. One of the largest customers for the API team is the mobile application development department, which constantly needs something for job seekers and employer applications.

For customers and developers to find a common language, you need a certain interface. We have it in the form of a project manager (MP). When the customer has a need for a new feature, he comes to the MP. Together they figure out the business requirements. After it became clear that the customer needs, the MP sets up a task in Jira, which brings in all the basic requirements. Such tasks are called portfolios, they all have the same type in Jira and a common set of statuses used to promote a new feature from idea to its realization.

Collecting and identifying requirements is a very important step. The customer and the manager should understand each other well, and it is equally important that both are well-versed in the area that will be discussed. Sometimes it makes sense at the stage of forming an idea to connect to the issue of developers as consultants in order to avoid technical problems in the future.

Stage Two: Planning

We have a recorded idea from the customer. At this stage, a clear task for developers is required. Find out how difficult it is to implement the task as a whole, to determine the key approaches to solving the problem. Find the pitfalls. Sometimes it can be very expensive to implement an idea, so you have to find a certain balance between complexity and requirements. At this stage a large number of people may be involved, until all agree. If a task requires a complex or ambiguous architectural solution, then it is often taken out on the AB (Architecture Board). At hh.ru, this is a weekly meeting of timblids with a technical director to solve and discuss pressing problems and sharing knowledge between teams.

Sometimes it becomes clear that the task is too large or contains several tasks. Then it can be divided into separate portfolios.

Often, when implementing a new idea, it makes sense to first make MVP (minimum viable product) - the minimum viable product that will have only the minimum number of functions sufficient for the life of the product. So you can check how this product will go and whether to develop it at all, or make a working version in a short time if the full version is done for too long.

Stage Three: Decomposition and Evaluation

Decomposition

When it becomes clear what a task is in technical terms, it can be decomposed - divided into stages and subtasks. Most often, one of the developers deals with the decomposition. He conducts more detailed portfolio research from the technical side. Clarifies incomprehensible moments from code maintainers with whom you have to work. For each subtask, a task is created in jire. Each task is a separate task, a change that can be rolled out for sale - and nothing will break. What are the tasks? We have an API team, if the interface is expanded — and, ideally, by the way, it can only expand so as not to violate backward compatibility — then the task is set up to refine the documentation.

Imagine that we are issuing some new data to the API. Then we may still need a task to issue this data in the API application, and in order to drag this data out of the backend, we need to create a task to refine the integration layer, and we also need to make the correct data output in the backend itself - this also requires a separate task. Changing the functionality should be accompanied by writing auto-tests, which are run before the output of tasks in production, in order to increase the confidence that bad code will not get into the work. For autotests, of course, also need a task. All of these tasks are blocked on top of each other in dir in the right order so that they can get out in the right order. Also, tasks are tied to the corresponding portfolio.

Decomposition is a time consuming process. It is he who determines the stages of work on a new feature. It requires knowledge of the architecture of the system being developed and an understanding of the processes taking place. At this stage, it is often necessary to analyze the existing code in order to understand where to embed the new one.

Evaluation

When the decomposition is complete, the whole team gets together to play poker. In planning poker. Its meaning is to assess the complexity of the task. Now I call tasks not a separate feature that we do, but its subtasks, which can be released separately. Each task from the portfolio is taken, a bit is discussed with all developers, so that everyone can understand what it will have to do in it, and then each gives an assessment of the complexity of this task, choosing a special card with the desired value. Then everyone opens the cards at once - and ideally everyone should get the same score. If someone stands out from the majority opinion, he must argue his position. After listening to the arguments and counter arguments, you can repeat one more visit or come to a common opinion. As a result, its complexity is recorded in the task. This is necessary in order to predict the execution time of the portfolio as a whole.

Previously, we gave estimates of tasks at story points (sp). 1sp is one perfect working man-day, i.e., a rough idea, how much you will do this task, if you sit down, you will be engaged only in it and no one and nothing will distract you. But the assessment in sp turned out to be not very convenient for us, since it is fairly accurate, which required a certain amount of time for each task for everyone and the assessment was sometimes delayed. Then we decided to evaluate in T-shirts.

Greater accuracy of portfolio evaluation was not required there. Moreover, the marks given in storypoints were also not always accurate and floated slightly up and down. So what are these T-shirts? T-shirts are a common practice in agile. Usually it is used as a transition to storipointam. We did the opposite. When evaluating, they operate not on the T-shirts themselves, but on the size. We have defined such gradation for ourselves. To which description the task is most suitable, then we assign it.

• 0 - do not do anything;
• XS is nonsense, you can do it very quickly;
• S is a simple task;
• M is a common task, everything in it is clear;
• L - difficult, there are incomprehensible places;
• XL is complex, little is clear. Such tasks should not be. Either it is necessary to send it for additional research, or it should be divided into subtasks;
• XXL - this jersey was added to jira, but I don’t know why - there shouldn't be such people either. This task is like a whole portfolio, probably.

There are many applications for phones in the game of planning-poker with jerseys and sidepoints.

After all the tasks are evaluated, you can sum up the T-shirts. Each t-shirt is assigned a number reflecting complexity. The sum of these numbers determines the approximate complexity of the problem. It is not so important what these numbers are equal to, they do not reflect the real time of portfolio execution, but by them it is possible to approximately understand how much one portfolio is heavier than another. This estimate is also used to calculate the technical tax. Teams have to spend 30% of their time on it. Technical tax is a task that does not belong to any team and is usually aimed at improving something. For example, cutting out old code, refactoring, switching to new libraries, increasing stability / performance, adding new amenities, etc.

Stage Four: Development

Stand up

Before telling about the development itself, I want to talk about such a useful process as a stand-up (or scheduling, if you will) accompanying the development. Every working day the team is going to negotiate. We have a list of topics that we discuss daily and 30 minutes of time for this discussion. Ahead is always the most pressing topic: we go through the portfolios, starting with the most ready ones, and find out what else is needed to release this portfolio, and we are trying to put all our forces on it. Also at stand-ups we discuss the problems we encounter in our work, we share important information. In general, you can talk a lot about communications in hh.ru, this is a topic for a separate post.

I just want to say one more thing: do not be afraid to ask questions. Even if they seem stupid and elementary to you. For example, I sometimes come across questions that I don’t know the answer to, but at the same time I think everyone should know it, and I’m ashamed to ask. But it is much better to ask and find out how to continue to be ignorant. In addition, others actually rarely think that they are being asked stupid questions, but rather, on the contrary, sometimes a person is even pleased to show his competence. There are, of course, exceptions with all sorts of annoying, not very sane or clever people, but I hope that the reader does not fall into this category.

Task selection

We gradually comprehend the Kanban philosophy, one of the rules of which states that it is necessary to reduce the number of portfolios that are simultaneously in operation. Tasks sometimes bloom on each other and the next one cannot be taken until the previous one is ready. Or, sometimes, the portfolio is suspended, because it is waiting for some other problems to be solved. Because of this, we usually have 2-4 portfolios at the same time in our work. There are 4 developers in our team now and we are looking for the fifth one (blink). The team welcomes cross-functionality. Any of us can choose from a portfolio a task in Java, Python, or take a task for autotests or documentation. You can take any task from the portfolios that you like, but there are some loose rules. For example, if the portfolio is an extension of the API interface, then the first task is to perform the documentation task, since at the documentation stage a clearer picture of how the interface will look will appear, and all the rules will be taken into account. Having the documentation is easier to develop. The sequence of tasks is dictated by the priority of portfolios, and of course by common sense - it is easier to take a task in the frontend, when data is ready for it on the backend, but such tasks can often be done in parallel.

Development

We chose a task. Finally, you can start developing! Open your favorite IDE (thank maven) and start writing code. There is plenty of room for testing. Each developer has his own test bench. On the stand there is an environment very close to the production environment. The easiest way to test a service is to run it locally. Service settings allow you to send requests to the test bench to interact with other services. Sometimes this is not enough and you need to start the service in the context of other services, for example, to correct the backend and pull the necessary pages of the site. To do this, you can close the service from the git branch to the stand. Code taken cover unit tests. They are chased away before the deployment to the stand, and if something went wrong, we will find out about it.

About the development I will not say anything. And so the whole bulk. Just do your job well, so that your conscience is clear, but do not get too carried away to keep up with the deadline.

Review

The obligatory part after making changes to the code is a review. Review helps to fumble knowledge between developers and improve the quality of the code. How I choose a reviewer. If something complicated rules, then it is better to attribute the code to the person who best understands it. Tasks from the API service are usually reviewed by API members - it's good to know who does what on the team, plus I like the fact that we have a review in place. Services have maintainers and other caring people who signed up for the project on the githaba, some of them like to look at pull-requests, even if they are not asked, for which they are thankful. In general, it is worth revising as soon as possible so that the task does not hang for this reason. The review usually does not take much time, and the tasks have to be pushed forward.

Autotesting

Another mandatory part of the release task is self-testing. This is necessary in order to check the performance of the main places after the changes. This is not a silver bullet, but we have good test coverage and it saves a lot of problems. In order to get rid of autotests, you need to set the changed service on the stand and run the job in jenkins. TestNg-tests will be launched, which will go through the main interfaces of the site, API and other places and check the correctness of their work.

Release

After the autotests have passed, the task can be transferred to the jire in the status Ready to release. All tasks from one service with this status and not blocked by other tasks can be released at one time. There are services for which many tasks are issued. These are released every day. The rest are available as needed. I came to HeadHunter two years ago, then I had to take a lot of actions to release a release. There was a long instruction. Before I came, they say, it was even more difficult. Various teams took turns answering daily releases. The team usually selected one person in charge. His task was to perform actions from the instructions, deal with problems if they arose during the release process, and ensure that everything was good. To date, we have made great progress in the release of releases, which are now going out automatically and human intervention is not required much, but nevertheless the teams continue to be on duty with releases. In short, when released, the following happens:
it is necessary to update the release stand to the current state ';
collect in the git-branch of the release-candidate, where all issued tasks will be merged;
stop the service from the release-candidate on the test stand;
run autotests;
close the release-candidate to the prod. This stage takes place separately by means of the operational service;
hold release-candidate to master.

Machine works well. Now you need to intervene only if something went wrong. For example, it may be that the release stand was not updated for some reason, or there was a conflict when building a release candidate or, for example, not passed autotests. If a task is released and something still goes wrong, you can always quickly roll back to the previous working version.

A / B testing

Some teams, before releasing a feature in life, conduct A / B testing. The feature is released to a separate group of users, it works for some time, data is collected, and at the end it compares who showed the best results - a new feature or what it was before. If the new feature won, then it has the right to life. All, of course, a little more complicated .

Slack

Separately, I want to note the participation of Slack in the development process and communications in general. There, of course, is full of flooding, sometimes it is distracting and you have to teach yourself how to use it correctly so that it brings more benefits. Firstly, there you can always ask competent people a question, especially if you work remotely, secondly, this is a convenient place for sharing knowledge, documentation, etc. Thirdly, connecting all kinds of things to it: for example, receiving notifications that the stand has been updated , events from the calendar, alerts from the monitoring system and much more.

Stage Five: Demonstration and Feedback

Finally, when all the portfolio tasks are released, the task assigned to the portfolio is completed, you can brag about what happened. The customer, other interested persons and optionally everyone are invited, and a small presentation on the work done and answers to questions are given. After the portfolio in Jira is transferred to the customer, so that he left feedback about the work done. This completes the life cycle of the portfolio.

By the way, the demonstration is sometimes held before production release, in order to collect feedback, opinions, questions and suggestions, share, boast and just understand that the movement is going in the right direction.

And more?

And I also want to say, although the observant reader has already noticed that the life cycle of portfolios and tasks is managed in Jira. They have different statuses, which are tied to all sorts of processes. Also, these statuses are useful in that you can, for example, look at the statistics / dynamics of an individual team or of all teams as a whole, draw conclusions, in which places the gaps, and maybe something is possible to improve there. We use kanban boards to display portfolios and tasks.

This is how difficult and thorny the path to life is. On this way, many tests await her, but this is a fully justified way. As a result, we get a strong and well-formed feature that can withstand all the tests.

In conclusion, I want to say that it is not necessary to arrange a cargo cult from the process. We need to realize what is beneficial and what is not. And do not change everything at once. But we must try, experiment, choose the best, gradually change.