The main thing is the tail. Software development pipeline technology

Written in collaboration with Revaz Bukhradze (Editor: Angelina Kipelova)

The technology differs from the handicraft industry in that the results are repeatable, and the time frame for their achievement is predictable, as well as in any science, the results of the experiment are recognized only if they were able to be repeated (and even better - put on stream). And its meaning is to successfully reproduce the algorithm of work every time it is needed. For example (this is just a bad example) Chinese printing houses print catalogs and packages cheaply, but they need an eye and an eye. Today, they used customer-defined settings, and tomorrow they decided that they knew which cheaper better colors to use. And, say, instead of black and yellow stripes Beeline can find a red-blue color scheme. And the eastern team of workers, left unattended in the process of laying a brick wall, can depict the installation "traveling wave" with the help of improvised building materials. That is, the sense of adaptability in the fact that the system worked autonomously, and the result at the output each time was equally successful. Well, with rare inevitable exceptions.

Philosopher's Stone of Universal Methodology

Experts argue to bloody holivar , which software production model is better, and when (which and for what) the methodology should be applied in large companies. Where it is impossible to keep track of the development department physically, and where any delays and critical errors sensitively affect the work of the giant colossus. I really want every soldier to become a general to a specialist by feeling the same “universal” method, which will reduce the number of problems in the development process, while increasing efficiency to the level “wipe, competitors!” (However, as you know, there are no silver bullets ).

However, if you do not determine what this “best solution” means in each specific case, the answers found will be similar to furniture from Ikea: “these are general recipes, and then all by yourself”. Just as the medicine is chosen individually for a number of parameters (although the result of the striving is the same everywhere - recovery), the software development methodology also implies a preliminary analysis of the “given” field.

Using different software development models: when and what works best

• The waterfall model is effective if the requirements are well defined, and they will not change much during the project. The customer will not come running, waving fresh TK, and will not flop it on the table with a loud statement: "The concept has changed!". If the deadline does not blaze, and the resources of good qualifications are sufficient for the project, the model is undeniably beautiful.

  
  

• Iterative - suitable if you expect a change in the concept still stands. But not the fact that the customer will decide on drastic measures. With a fairly stable skeleton concept, it is possible to modify the details of the project. In such a model, the possibility of realizing the final result “by parts” is allowed. At the same time, it is very important that resources not only have good qualifications, but also have a broad profile that adapts to changing requirements.

  
  
• Agile is a model loyal to the absence of certain requirements at the beginning of a project. Looks like “Boris! Fire the fish! ”“ But where is the fish, Stepan? ”“ Borya, for now, start, and the fish will be! ”. That is, if the requirements are vague, and will appear, change and turn upside down.
  This model allows you to implement the final result "in parts" with the earliest receipt of a working product that implements the main idea. The requirement for this model is a sufficient amount of resources with good qualifications, and an essential part is a broad profile. Adaptation to changing requirements should be instant.

There are other methodologies [ 1 ], [ 2 ] but the ones listed above are considered basic and are widely used in practice.

Why so?

This list is not taken from the air of St. Isaac's Cathedral, where it was inscribed with the wave of the Foucault pendulum. If we analyze where the recommendations came from, we can see that the main decision-making drivers when choosing a methodology are:

• the level of uncertainty and the risks associated with errors in the early stages of the process (collection and analysis of requirements, design);
• staff availability and competence;
• time to get the finished result.

At the same time, why would the seemingly inefficient waterfall model, which very strictly limits the variation of requirements, is alive, well and actively used in practice? In addition to dealing with uncertainty, an equally important factor is ...

Production economics

If the task is to minimize production costs, the waterfall model (under its own conditions) demonstrates confident working efficiency. However, the key word here is “condition”. Therefore, for each of the methodologies presented above, you should consider a diagram of the processes and their costs.

We will conduct a qualitative comparison of the economics of the processes for the waterfall model and agile, taking $$$X$- the labor required to implement a given amount of functionality, and for $$$C$- labor costs associated with the production cycle. In this case, the total workload will be:

• for waterfall model $$$(1 + p) \ cdot (X + C)$, here $$$p$- the coefficient associated with the volume of improvements resulting from changes in the initial requirements;
• for agile - $$$(1 + p) \ cdot X + N \ cdot C$where $$$N$- the number of iterations required to achieve a result.

Comparing these results, we get the following conclusion:

• waterfall model agile more efficient in case $$$(1 + p) <N$that is, the volume of processing is small.
• with a high degree of uncertainty and a high risk of improvements $$$(1 + p) \ ge N$agile becomes more efficient.

You may also notice that the automation of production processes (or trendy DevOps) significantly increases the economic efficiency of agile by minimizing costs (associated with the length of the production cycle), because these production costs $$$C$it's much easier to manage than customer desires and fantasies $$$p$.

It should be emphasized that the above reasoning is qualitative, not quantitative. Therefore, in each particular case, it is necessary to do the calculation based on the current distribution of the roles of production employees in the company.

The economic aspect on this can be considered closed, the selection criteria are quite objective. Now go to the next question.

Resource download

At that time, the planned economy got burned, when the same result had to be obtained under different working conditions. For example, Astrakhan and Eskimo tasked to grow a ton of watermelons. Take out and lay out - by August should be! The first is just to pick up a shotgun, loaded with salt, and chase the Darmovshchina hunters from the melon. And the second will have to invent and equip a high-tech microclimate system.

The issue of resource loading is best considered by the example of participants' profiles of the processes mentioned above. For research we will use the iterative RUP approach, and we will see uneven downloads (picture from wikipedia).

These irregularities require management dancing with tambourines noticeable effort to meet the deadlines for the project. It is necessary to minimize downtime and give customers real deadlines for completing tasks. Moreover, for the waterfall model and for agile (in terms of the sprint), in view of the fact that the stages within the respective production cycles are organized sequentially, this problem with uneven use is more acute than for models of an iterative type. Why is it sharper? We will build a role profile for, for example, for waterfall - a development cycle of 10 weeks.

Now suppose that we have the next task similar in size and complexity to our approach and we need to plan its optimal implementation. If you do not defragment the download, or how often you joke with Tetris, you’ll get the following picture, which allows you to release the finished product once every 7 weeks with a 10-week production cycle (the gap between releases is marked with a red arrow <=> ):

which, due to the presence of a “gray zone” and its rather large size, is obviously not the optimal solution to ensure uniform loading of personnel.

Now pay attention to the gray area - it is large enough to make a clear conclusion: the calculation is not optimal. There is no uniform loading of personnel here. However, after some optimization and redevelopment, the picture can be somewhat improved: to reduce the total duration of projects by two weeks, to optimize staff loading, which, with the same 10-week cycle, will allow to produce the finished product once every 5 weeks:

Conveyor problem - just in time

It is only on paper that it turns out to perfectly optimize work. And de facto at the time of planned downtime, it is quite difficult to figure out how to occupy resources. In theory - to give a useful load, for the corresponding periods having given them, for example, to other projects. But for this we need projects that are implemented in the "pieces". That in reality, rather, utopia than a real possibility. Most projects need to do "from and to". Therefore, in the full-fledged periods, allowing to achieve a result, the holes are gaping, and the management has to re-plan again and again. To create trawls, periods of processing and regular downtime. The same problem, only in a smaller volume, is also characteristic of an iterative approach - it suffices to take as a unit of measurement not a week, but 2-3 working days.

Separately, it is worth noting that this problem is also characteristic of the agile methodology, and in view of the short duration of the sprint (usually two weeks), it is still aggravated. To solve a complex task that requires significant labor-intensiveness, it is necessary to go beyond the limits of the workload of one sprint. Consequently, the very “rush job” appears: the load on analysts and architects increases in terms of splitting the task into subtasks and maintaining the integrity of the result of each sprint.

We will formulate what we want from the ideal project implementation process, and what criteria it should meet:

1. have reasonable administration costs;
2. maintain a high degree of certainty of requirements, while giving the customer residual freedom of imagination;
3. to provide substantial workload for each production cycle, allowing you to solve complex tasks in one step;
4. guarantee the regular release of the product in order to be able to pay for the work of the team;
5. ensure continuous and uniform loading of the team.

And now we will try to refine the iterative methodology in such a way as to meet the criteria that we have defined for ourselves.

And here Kanban?

The famous Japanese methodology is used not only at the Toyota plant. If it is used in iterative development, it allows you to create an analogue of excess stocks in the production model and synchronize the relative amounts of work performed at various stages of the production cycle using the specified restrictions on "work in production" (WIP - work in progress). Thus, work in progress is minimized. At the same time, the process is quite economical, while the work capacity per cycle can be increased. However, the criteria for regular release and guaranteed load are not met by default.

To ensure their implementation, it is necessary to compare the production cycles of the assembly of machines and software production (images found on the Internet).

Toyota conveyor	Development office

Feel the difference?

And it is in the fact that the processes on the pipeline are built in such a way that the duration of each operation is approximately the same. The algorithm is known, tested and automated. Therefore, there is no downtime or accumulation of parts moving along the conveyor. Moreover, the conveyor is designed to move evenly and continuously.

Nevertheless, the kanban system can be tried on in software production, if you do the same, you can set up a pipeline. That is, to determine the duration of the stages of the production cycle and try to withstand them.

The duration of the production cycle will be equal to $$$16$weeks, while the finished product will appear once a $$$4$weeks, which will allow us to gradually “catch up and overtake” the classical iterative model, namely, if we compare the duration of production cycles in weeks, we get $$$10 + 5x = 16 + 4x$where do we get $$$x = 6$i.e. on the seventh production cycle, we will already get the effect.

This approach to construction provides a uniform load of all the resources involved, but in order to go to it, it is necessary to take into account an important subtlety. It is related to the number of personnel involved, necessary to ensure the continuity and uniformity of the process.

The number at each stage must correspond to the amount of work received from the previous stage. That is, everything that came, should be processed on time, and what did not work out to master will be transferred to the next production cycle. Therefore, in order to move from waterfall or classical iterative development to an innovation system, it will be necessary to rebalance the existing number between stages.

To do this, we first introduce the unit of measurement of the available labor capacity - let it be a “square” or square. This unit corresponds to the volume that a command can make from $$$n$man for a week. Then balancing should be carried out as follows:

• Analysts and designers - the amount of available labor in the iterative development of 10 sq. M., The effective labor volume of 4 sq. accordingly, with a 4-week production cycle, 4 square meters are needed. The number of analysts and architects who need to be reoriented to other works is 60% of the current number;
• Developers and technical writers — both in iterative development and conveyor design — 12 square meters each, and all the difficulty is effective — rebalancing is not required;
• Testers - with the iterative development of 10kV work capacity, with effective - 7 sq.; at conveyor - 8 square meters. That is, reorienting to other projects requires 20% of the current number by role;
• Implementers - the amount of available labor in an iterative development of 10 sq. M., The volume of effective labor capacity of 3 sq., In conveyor development - 4 sq., I.e. the number of implementers who need to be reoriented to other works, is 70% of the current number.

However, this approach, like everything else in IT, is not a “silver bullet”, killing a deadly monster on the spot. Yes, the methodology allows for a good solution of issues related to the optimization of resource planning, but for the tasks it is necessary to take into account the risks directly following from the rules for constructing a pipeline:

• At the stages of analysis and design it is necessary to take into account the workmanship available at all subsequent stages;
• With the conveyor scheme, there are only minimal opportunities for upgrading, and completions and other work on “tailing”. All that does not pass the appropriate stage, in time should be removed from the conveyor, as a marriage. And these indicators must be taken into account in the amount of improvements on the next cycle. Otherwise, the pipeline will be out of sync and immediately lose its effectiveness (and with it the meaning);
• A conveyor is by its nature, providing high efficiency, is in some sense a sweatshop;
• Unlike aglile's “fun-filled” approach, which stimulates the expansion of competences in all areas of the team to ensure interchangeability, the pipeline is aimed at deepening competencies and enhancing personal skills in its and related fields. That is, the developer must more or less understand the architecture in order not to even begin coding according to the specification, which he considers to be of poor quality.

Consideration of these risks requires tougher planning at the initial stage and due courage in making decisions about refusing modifications at the last moment, when you really want to do this, and the deadlines, as usual, are tight.

findings

, , , — , , ( , F1 agile). , .