Badoo development process

Today we will conduct a tour of the development department of Badoo, in which a new functionality of our site is created, we will tell you about the process itself - from setting the task to the moment it is displayed in the combat environment.

During the existence of the company, its workflow has evolved from stochastic to well-controlled and understandable. One and a half or two years ago, no one could exactly answer the question of which tasks were in today's calculation, and the manager interviewed the programmers who laid out what. Now the process is as automated as possible, the task tracker is closely connected with the version control system, the tasks go through several stages of testing. At the same time, high development speed was preserved: the situation when the task appeared in the morning is considered normal, and after lunch the new functionality is already available to users on the site.

Main steps

The whole process can be divided into stages:

1. Work on the product: the idea, the preparation of TK.
2. Technical implementation: decomposition, development.
3. Testing and translations in different languages.
4. Release for some users.
5. Release for all.

As you know, any process rests on the people involved in it. Let's calculate how many people participate in the typical case of the process described above: product manager, designer, copywriter, project manager, developer, layout designer, developer reviewer, QA engineer. This list lacks one actor. Who we intentionally did not mention? Write your version in the comments!
')
Nine people is the minimum number of participants in the process when we develop new functionality in one language only. This is usually a test release for a specific user group or a special promotion for a specific country. When we make a release for all users, the process also includes translators who translate into 40 languages. Some languages ​​are supported in several dialects, for example, American English and British English.

The whole process is controlled and formalized using a well-designed workflow in the task tracker (we use JIRA, www.atlassian.com/software/jira ).

Work on the product

It all starts with an idea. Usually, it appears either as part of the “How to increase such an indicator” research, or as an improvement to the existing functionality. The development of the idea is entrusted to a specific product manager, for example, who is responsible only for postal and push notifications, who draws it up as a task for his team with the status of “Idea”.

Accumulated ideas are periodically reviewed and a detailed study of the most important of them begins. At this point, the task is transferred to the “PRD Preparation” status, after which the product manager prepares a short description of the proposed addition and sends it to the designer and copywriter. More precisely, the task in the task tracker is transferred to them, and all those who are interested subscribe to its updates in order to be aware of its current status.

PRD (Product Requirements Document) is a cross between a technical task and a sufficient description of the new functionality. Usually, the final PRD gives an overview of how it will look and how it should work. All implementation details are specified during the development process.

As soon as the design and texts are ready, the manager collects all the elements (layouts, text files) in the PRD and completes the detailed description of the new service. As a result, when the manager believes that his task becomes as clear and autonomous as possible, she is assigned the status “Ready for development” and is transferred to the project manager from the Features Team. Often, the task is not accepted from the first due to an incomplete description, a large technical complexity, or other nuances that are discovered when it is checked by the project manager.

Prioritization of the task and the performer

The priority of the tasks sent to the development is determined by the general manager of the Product Team. In this case, preference is given to those of them that are an integral part of large-scale projects to change certain parts of the site. The importance of infrastructure tasks (bugs, research, etc.) is determined by the developers themselves.

In turn, the developers put the correction of critical bugs and the minimization of errors in the logs in the first place - for each QA-department it gets a new task with high priority. Moreover, the monitoring team monitors the state of the site around the clock and can contact the developer during off-hours to fix a broken important component.

Creating a new functional is engaged in a special team of PHP-programmers, which is divided into groups of 2-4 people. Each group is responsible for its own set of components. Since the task is necessarily attached to a specific component, the group that will perform it is automatically determined.

All developers see their tasks on a dashboard (an English dashboard - a page with a set of different filters for tasks, set up specifically for each group) in the task tracker, where they are grouped and sorted in a certain way. There is a pool of tasks per group and there is a list of tasks to which the developer is directly related (author, performer, reviewer). A situation where a person has nothing to do is impossible in principle. Below is a schedule of tasks set and completed from task tracker for November, from which it is clear that the work really boils!



Below is a schedule of tasks in production. Using it, you can estimate the daily volume of fixes and improvements in Badoo:

Technical implementation

From the moment of development and up to the calculation of the combat servers, the task, regardless of its complexity, is fully led by one PHP programmer. If necessary, he puts a subtask on other components to colleagues from his department, and also collaborates with other specialists (layout designers or system programmers). If the task is non-trivial and requires specific architectural solutions, a working group of several of the most experienced developers is assembled, who together determine what will be done and how.

Consider this step on the example of creating a photo-rating ( badoo.com/vote/ ), the new version of which was recently launched. From the Product Team, developers received a document containing the following items:

• general description, project objectives (for example, to encourage users to upload new photos);
• a detailed description of the nuances (how the rating of the photos is calculated, how the following photos are selected, etc.);
• Layouts of interface elements of all necessary pages and blocks with a description of what each button means, what should happen when clicking on links, where and which menu items should be added;
• Description of what the unauthorized version of the service looks like.

Based on this document, tasks were assigned to different departments: layout of templates, development of C services and infrastructure on the site (interface elements, new blocks, interaction with services, etc.).

Verification of each completed task is rather formalized: there is a certain checklist and an automated process. Each task is developed in its branch of the version control system (we use Git, git-scm.com ). The branches have special naming rules ( example: BFG-123_another_cool_ticket), all commits automatically get the task number at the beginning of the description ( example: [BFG-123]: great improvement on photo rating) - this helps to track the movement of the code from the developer to build and final hitting the master branch.

Each time changes are sent from the local copy to the general repository, hooks are triggered, which check the formatting and formatting of the code. To format the code for our internal standards, a special phpcf utility was created that can both check and correctly format the code (manually, only those sections of code that, according to the developer, are more logical to rewrite than leave in the form of cumbersome but properly formatted construction).

After the developer completes the development of the task, it gets to the code-review person responsible or having a good understanding of the component. To make a review, we finalized the gitphp utility ( www.gitphp.org/ ) so that it became possible to view the general diff branches, leave comments on the code, which are subsequently sent as a single comment to the task in JIRA. Usually the reviewer checks whether the task is implemented in accordance with the rules adopted by the company, whether there are any obvious errors in the form of abandoned debag; may indicate the feasibility of using another design pattern or suggest a more optimal implementation.

The task must necessarily include unit tests for the changes made, while existing tests should not be broken (!). After completion, all tests are run on the appropriate branch using TeamCity ( www.jetbrains.com/teamcity/ ), and if some of them fail, the developer will receive a notification about this. For a number of tasks, monitoring of important indicators is pre-configured. After checking by the reviewer, the task falls into the quality control department.

Testing and translation

The completed task falls into the pool of the quality control team and is assigned to any free tester. Usually there are no responsible for the components, everything is testing everything, but for particularly important or urgent tasks, developers can agree with the testing department and transfer the task to a more experienced tester in a certain direction.

It is considered a good practice for a developer to write instructions on how a completed task works, and what needs to be tested. Best practices for testing critical components are described on the wiki (we conduct all similar documentation using Confluence, www.atlassian.com/software/confluence ).

During testing, all new and modified words and phrases from the templates fall into a special translation system (the developer can start this process in advance). From it, when displaying templates are generated for each of the supported languages.

To test tasks, our infrastructure supports several possible environments:

• local domains on development servers that emulate the combat infrastructure. Each developer conducts work on his domain, and the tester checks the tasks on his own;
• individual test combat environment: you can simultaneously check an unlimited number of tasks independently of each other;
• the general test combat environment in which the build gets is a set of tasks ready for display. In such a set, tasks are tested in complex, so that one does not break the other.

When a task receives the status “Tested”, it will automatically fall into the next build (if there are no unfinished child tasks and all translations are ready). Of course, conflicts are resolved manually - an alert comes about it. Non-working tasks are rolled back and the build leaves without them.

Release

After all the checks, the task along with the build is decomposed on the combat servers. This happens 2 times a day (morning and evening). Often, it is required to test a new functionality on a small group of real users, to evaluate the efficiency and load, before enabling it for the entire site. Usually the inclusion is done for a few countries or only for the development office. Most tasks are developed immediately with the possibility of partial inclusion and fast rollback.

But sometimes after the calculation it happens that our innovation does not work. Suppose that an unverified code was quickly laid out, or somewhere a logical error was made. Even the most experienced developers sometimes make mistakes that lead to the inoperability of some parts of the system, and sometimes it even takes shape according to the principle of dominoes.

Each of these incidents is composed of a “post-mortem” —a document that details the causes of the problem, what it affected, how quickly it was corrected, and, most importantly, what was done to prevent such a problem from occurring. After this, the post-mortem is put up for general discussion and everyone can speak out and express their vision of the situation.

In order to minimize such incidents, we try to automate and improve as much as possible our development processes, code management, control over calculations, as well as monitoring tools.

Total

The result of a lot of discussion, automation, the introduction of preventive measures, the constant audit of our systems is that now the whole development process in Badoo is understandable, effective and, quite importantly, fairly safe. Of course, it is impossible to exclude all possible errors, but I, as a developer, feel comfortable and confident when I edit and develop a new code, knowing that it will be repeatedly checked. On the one hand, it is somewhat relaxing, but on the other hand, it is possible that a programmer should feel that way in order to really devote himself not only to development, but to the creative process?

Alexander treg Treger, developer.
Andrey uni Sas, manager of offline notifications.