Agile is dead, long live ... Agile
Over the past few years, flexible methodologies have almost supplanted traditional development methods — two-thirds of IT companies are now working entirely on the principles of Agile. Did the expectations come true, what problems arise and where does everything go? We offer an analysis of the existing Russian and foreign experience in Agile and answers to these questions.
Despite the fact that Agile appeared about 20 years ago, it began to be used more or less actively only in the last eight years. Flexible principles emerged as an alternative to traditional development methods in order to reduce the cost of producing software that is ready to be sent to the customer (potentially shippable software) by increasing the efficiency of collaboration and customer focus. That is, a set of principles was formed for solving business problems - to accelerate the development process and achieve maximum results from the team without increasing production costs.
Yes, a flexible methodology allows you to squeeze the maximum efficiency out of the team, but there are two nuances that, under the circumstances, reduce the Agile effect. First, the principles really work only in small teams. In addition, if all the specialists in the team are very strong, it’s not at all the fact that it will turn out to significantly increase their productivity.
The current development process can be divided into three stages: writing code, preparing it for launch and production.
')
Traditionally, the goal of Agile was the first stage of development - writing code. At the end of the distant 90s, this stage was considered as requiring improvement and major changes. At the same time, one of the most common implementations of Agile - Scrum became famous. In fairness, it is worth noting that Scrum was published in 1986, 15 years before it was adapted for Agile in 2001.
Scrum brings a sample of what you need to write, allows you to track the development process, identify shortcomings and contradictions in the initial stages. Without a technical assignment, the programmer does not know what kind of product should be the result. For example, if we are talking about Google or Facebook, then the developers of this or that service do not have any questions. They make the product for themselves - they themselves are avid users and understand what and how it should look and what needs to be done in order for the new functionality to be convenient. Another thing, for example, is the billing system. The programmer does not know of the huge number of nuances that must be taken into account for the correct operation of the service, both regulatory and marketing, such as various models of computational operations .
Agile pioneers were small companies that were not afraid to apply advanced technologies - they had nothing else to lose! For obvious reasons, writing code in such organizations constitutes the main part of the process (the preparation and launch processes are practically absent). The effect was awesome! However, after several years of using traditional Agile in more developed organizations, they began to notice that the problems associated with a reduction in the release of new products to the market did not disappear. Agile moved the congestion to the next stage - to prepare a new product for release. At the same time, in the middle of the first decade of the 2000s, the concept of Agile was expanded and became collective, in contrast to the initial value, focused only on the first stage of development.
In order to prepare the written code for the potential release, Continuous Delivery appeared. It is worth noting that the release does not necessarily occur immediately, but may be delayed, for example, to link to a specific date in accordance with business objectives. Such an approach allows achieving a well-coordinated work of the team, automating all the processes and increasing the speed of potential code sending to production. However, the quality of the product is guaranteed only through automatic tests, as acceptance checks do not cope with the speed of delivery of the code and are often used as a formal step. Of course, today such automatic tests of individual components, often written by the developers themselves, ensure the correct execution of the code. But neither the methodology itself, nor any tools that exist today do not ensure the correct operation of the code in a more complex production environment. In addition, it is not a fact that the code will perform the functions necessary for the customer and solve the tasks set.
For example, in July 2015, after the next software update, trading on the New York Stock Exchange was stopped. This failure shook the global economy, in particular, reduced by 1% the US stock index for the whole day, and also slowed Greece’s negotiations with its creditors for a whole week! Another vivid example is the renewal of the air ticket sale system at Delta. After the release of new software in production, the information in general has ceased to flow to the dispatch service. As a result, the airline canceled all flights and suffered significant financial and reputational losses. Undoubtedly, in both cases, many of the checks before the launch were completed and, nevertheless, many more would have been done if they took less resources and time.
The code quality assurance gap is very relevant today and it will gradually be filled. According to Gartner, in the next three years at least 75% of IT corporations will have to develop automated testing that integrates business processes. Certain hopes for solving the quality problem are placed on such technologies and methodologies as containers and BDD (behavior driven development) - Docker, Cucumber, and others.
At the moment, Agile principles are implemented in most companies, where it was possible to do this without any particular business risks. And everyone who was able to build new processes is generally satisfied with the results, for example: Google, Zara, Ikea. However, it is worth noting that these companies operate on the principle of "divide and rule" - that is, the work is divided into projects that are engaged in relatively small independent teams. In addition, according to Agile, these companies primarily operate a business unit (especially Zara and Ikea), and not just a development department. They changed not only the organization of the process, but also the business, so they naturally correspond to the Agile model.
But there are some companies where it is not yet possible to fully adapt flexible methodologies - there are either no results at all or they do not meet the expectations of the business. For example, ING and Citibank are considered the most advanced in the banking sector in terms of Agile adaptation. However, not all departments of these financial giants work according to flexible methodologies. On Agile, those units that are engaged in surface shells, mobile applications or other advanced developments have switched. In other words, departments that are not related to core business processes, the failure of which can jeopardize annual revenues and lead to the collapse of the entire corporation.
That is, in those areas where the price of a mistake is very high, companies are still afraid to change something. Yes, they understand that Waterfall has outlived itself and, working as before, you can at some point be far behind the competition and lose everything. They would be happy to switch to new methods of work, but so far the risks are much higher than the potential benefits from innovations. In addition, large investments are often needed to translate business into flexible methodologies and it’s far from a fact that everything will work well at once. This applies primarily to those companies where most business processes are built on software written decades ago in programming languages that are not as modular as Java, Scala, GO, etc.
But those companies that tried to implement Agile and did not get the expected results are more and more discontent. They were promised that after the implementation of the flexible methodology, the efficiency would increase, and the time-to-market and with it the production costs would be reduced. However, the expected effect does not arise, as the code goes faster and more and more problems appear at the advanced stages of development. Efficiency in some cases increases, but there is no noticeable reduction in costs. But most often there are kickbacks from production that hit time to achieve goals, hit time-to-market and are very expensive, as fixing production bugs requires much more development resources. The cost of an error increases not only because of late detection, but also because of the high speed of development. This effect can be compared with turbulence in streams when the flow velocity exceeds the maximum allowed for a given environment.
All this has led to the fact that lately there has been a great demand for tools that allow to solve these problems and still achieve the Agile effect.
Clouds have been very actively developed in the last decade and partially opened gateways for testing on a large scale. However, the problem has not been completely resolved - despite all the advantages, cloud technologies are relatively expensive. In addition, it is not always easy to build the necessary verification environments in the cloud. Today, on average, each developer needs 10 environments - so the price of the environment can be compared to the developer’s payroll budget.
As a result, other technologies began to appear - containerization, data and services virtualization. Their use in integral environments opens up even more gateways. Using these technologies on a single server, it is possible in a matter of seconds to raise hundreds of environments that mimic real, and the malfunctioning of certain services.
However, there is still an imbalance between the frequency of changes and the frequency of tests that allow you to check these changes. Thus, the bottleneck has moved from development teams and simplified inspection environments to integrated inspection environments. By the way, this process is confirmed by the appearance of a large number of startups that are engaged in solving the problem of organizing a software environment.
One of the central concepts of flexible and continuous methodologies is the so-called left shift, which means transferring responsibility as close as possible to the developer. And this is true - now we see that very many processes have shifted towards writing code. For example, the discussion of technical specifications, the first verification of the code and even the presentation of a working application are often performed in the presence or even directly by the developer.
In the future, this shift will move further to the left, even closer to the programmers - that is, all checks, including integral and stress, will be carried out extremely close to the source of changes and to the time of the change. This is due to the need for a balance between the frequency of changes and the frequency of tests.
In addition, this shift avoids the effect of turbulence, which can be detected when working with several hundred people who submit a code several times a day. The effect of turbulence lies in the fact that hundreds of changes accumulate in anticipation of environments and the lack of media affects the speed of their delivery. It occurs for two reasons. First, because of the lack of resources (in an attempt to do everything, people create a lot of unnecessary actions, and this slows down the overall process even more). And secondly, if out of a hundred changes a few dozen “fall” on tests, then this leads to an even greater shortage of environments, since after correction, you need to check everything again, and not just those who ruled. For example, changes in the configuration module can lead to a failure in the payment module and the failure of the entire application.
Thus, in order to increase the Agile effect, it is first necessary to implement two conditions:
1. The emergence of an opportunity to run business tests at no extra cost,
2. shift the integral and business testing as close as possible to the source of change.
And, of course, we need effective solutions for monitoring application performance in real conditions.
Since its inception, Agile has reincarnated from a set of principles into an assortment of methodologies, processes, and even standards. Today the field of activity of these methodologies is not limited to the development teams. Flexible processes are successfully implemented in almost all IT departments and even business is guided by Agile standards. Among the most famous are Scrum, Scrumban, SAFe, ScaleAgile @ Spotify, Continuous Delivery, Lean, Prince2 Agile and many others.
Despite the many frameworks and methodologies that, within Agile, claim leadership and declare that all problems are solved and all barriers are removed, quality remains the main gateway to unquestioning translation into Agile of all departments and development stages of IT corporations. The existence of this gateway is causing distrust of Agile and an attempt to justify the preservation of the principles of the cascade model or Waterfall. Attempts are also being made to change Agile by introducing the traditional stages of testing.
This will keep for a time the flow of errors flooding business processes, websites and corporate applications. However, in conditions of tough competition, business will not tolerate such a model for a long time. Only when translating the whole mechanism into flexible methodologies is it possible to achieve a business that is flexible and adapted to the requirements of the market, with high quality and speed parameters.
So, one of the key problems in the work of Agile today is the problem of quality. And its decision is a matter of the very near future. So, a number of tools have already appeared that help to quickly and accurately predict the degree of business satisfaction with new changes in applications. The introduction of such tools allows to achieve good results in the implementation of inspections.
Having solved this problem, in the next 5 years flexible methodologies will open their way to the remaining third of the corporations, which until then do not dare to translate the development into a more modern, but risky way.
Despite the fact that Agile appeared about 20 years ago, it began to be used more or less actively only in the last eight years. Flexible principles emerged as an alternative to traditional development methods in order to reduce the cost of producing software that is ready to be sent to the customer (potentially shippable software) by increasing the efficiency of collaboration and customer focus. That is, a set of principles was formed for solving business problems - to accelerate the development process and achieve maximum results from the team without increasing production costs.
Yes, a flexible methodology allows you to squeeze the maximum efficiency out of the team, but there are two nuances that, under the circumstances, reduce the Agile effect. First, the principles really work only in small teams. In addition, if all the specialists in the team are very strong, it’s not at all the fact that it will turn out to significantly increase their productivity.
How it works?
The current development process can be divided into three stages: writing code, preparing it for launch and production.
')
Traditionally, the goal of Agile was the first stage of development - writing code. At the end of the distant 90s, this stage was considered as requiring improvement and major changes. At the same time, one of the most common implementations of Agile - Scrum became famous. In fairness, it is worth noting that Scrum was published in 1986, 15 years before it was adapted for Agile in 2001.
Scrum brings a sample of what you need to write, allows you to track the development process, identify shortcomings and contradictions in the initial stages. Without a technical assignment, the programmer does not know what kind of product should be the result. For example, if we are talking about Google or Facebook, then the developers of this or that service do not have any questions. They make the product for themselves - they themselves are avid users and understand what and how it should look and what needs to be done in order for the new functionality to be convenient. Another thing, for example, is the billing system. The programmer does not know of the huge number of nuances that must be taken into account for the correct operation of the service, both regulatory and marketing, such as various models of computational operations .
Agile pioneers were small companies that were not afraid to apply advanced technologies - they had nothing else to lose! For obvious reasons, writing code in such organizations constitutes the main part of the process (the preparation and launch processes are practically absent). The effect was awesome! However, after several years of using traditional Agile in more developed organizations, they began to notice that the problems associated with a reduction in the release of new products to the market did not disappear. Agile moved the congestion to the next stage - to prepare a new product for release. At the same time, in the middle of the first decade of the 2000s, the concept of Agile was expanded and became collective, in contrast to the initial value, focused only on the first stage of development.
In order to prepare the written code for the potential release, Continuous Delivery appeared. It is worth noting that the release does not necessarily occur immediately, but may be delayed, for example, to link to a specific date in accordance with business objectives. Such an approach allows achieving a well-coordinated work of the team, automating all the processes and increasing the speed of potential code sending to production. However, the quality of the product is guaranteed only through automatic tests, as acceptance checks do not cope with the speed of delivery of the code and are often used as a formal step. Of course, today such automatic tests of individual components, often written by the developers themselves, ensure the correct execution of the code. But neither the methodology itself, nor any tools that exist today do not ensure the correct operation of the code in a more complex production environment. In addition, it is not a fact that the code will perform the functions necessary for the customer and solve the tasks set.
For example, in July 2015, after the next software update, trading on the New York Stock Exchange was stopped. This failure shook the global economy, in particular, reduced by 1% the US stock index for the whole day, and also slowed Greece’s negotiations with its creditors for a whole week! Another vivid example is the renewal of the air ticket sale system at Delta. After the release of new software in production, the information in general has ceased to flow to the dispatch service. As a result, the airline canceled all flights and suffered significant financial and reputational losses. Undoubtedly, in both cases, many of the checks before the launch were completed and, nevertheless, many more would have been done if they took less resources and time.
The code quality assurance gap is very relevant today and it will gradually be filled. According to Gartner, in the next three years at least 75% of IT corporations will have to develop automated testing that integrates business processes. Certain hopes for solving the quality problem are placed on such technologies and methodologies as containers and BDD (behavior driven development) - Docker, Cucumber, and others.
Who does it work for?
At the moment, Agile principles are implemented in most companies, where it was possible to do this without any particular business risks. And everyone who was able to build new processes is generally satisfied with the results, for example: Google, Zara, Ikea. However, it is worth noting that these companies operate on the principle of "divide and rule" - that is, the work is divided into projects that are engaged in relatively small independent teams. In addition, according to Agile, these companies primarily operate a business unit (especially Zara and Ikea), and not just a development department. They changed not only the organization of the process, but also the business, so they naturally correspond to the Agile model.
But there are some companies where it is not yet possible to fully adapt flexible methodologies - there are either no results at all or they do not meet the expectations of the business. For example, ING and Citibank are considered the most advanced in the banking sector in terms of Agile adaptation. However, not all departments of these financial giants work according to flexible methodologies. On Agile, those units that are engaged in surface shells, mobile applications or other advanced developments have switched. In other words, departments that are not related to core business processes, the failure of which can jeopardize annual revenues and lead to the collapse of the entire corporation.
That is, in those areas where the price of a mistake is very high, companies are still afraid to change something. Yes, they understand that Waterfall has outlived itself and, working as before, you can at some point be far behind the competition and lose everything. They would be happy to switch to new methods of work, but so far the risks are much higher than the potential benefits from innovations. In addition, large investments are often needed to translate business into flexible methodologies and it’s far from a fact that everything will work well at once. This applies primarily to those companies where most business processes are built on software written decades ago in programming languages that are not as modular as Java, Scala, GO, etc.
But those companies that tried to implement Agile and did not get the expected results are more and more discontent. They were promised that after the implementation of the flexible methodology, the efficiency would increase, and the time-to-market and with it the production costs would be reduced. However, the expected effect does not arise, as the code goes faster and more and more problems appear at the advanced stages of development. Efficiency in some cases increases, but there is no noticeable reduction in costs. But most often there are kickbacks from production that hit time to achieve goals, hit time-to-market and are very expensive, as fixing production bugs requires much more development resources. The cost of an error increases not only because of late detection, but also because of the high speed of development. This effect can be compared with turbulence in streams when the flow velocity exceeds the maximum allowed for a given environment.
All this has led to the fact that lately there has been a great demand for tools that allow to solve these problems and still achieve the Agile effect.
How to increase efficiency?
Clouds have been very actively developed in the last decade and partially opened gateways for testing on a large scale. However, the problem has not been completely resolved - despite all the advantages, cloud technologies are relatively expensive. In addition, it is not always easy to build the necessary verification environments in the cloud. Today, on average, each developer needs 10 environments - so the price of the environment can be compared to the developer’s payroll budget.
As a result, other technologies began to appear - containerization, data and services virtualization. Their use in integral environments opens up even more gateways. Using these technologies on a single server, it is possible in a matter of seconds to raise hundreds of environments that mimic real, and the malfunctioning of certain services.
However, there is still an imbalance between the frequency of changes and the frequency of tests that allow you to check these changes. Thus, the bottleneck has moved from development teams and simplified inspection environments to integrated inspection environments. By the way, this process is confirmed by the appearance of a large number of startups that are engaged in solving the problem of organizing a software environment.
Where does Agile stand and go?
One of the central concepts of flexible and continuous methodologies is the so-called left shift, which means transferring responsibility as close as possible to the developer. And this is true - now we see that very many processes have shifted towards writing code. For example, the discussion of technical specifications, the first verification of the code and even the presentation of a working application are often performed in the presence or even directly by the developer.
In the future, this shift will move further to the left, even closer to the programmers - that is, all checks, including integral and stress, will be carried out extremely close to the source of changes and to the time of the change. This is due to the need for a balance between the frequency of changes and the frequency of tests.
In addition, this shift avoids the effect of turbulence, which can be detected when working with several hundred people who submit a code several times a day. The effect of turbulence lies in the fact that hundreds of changes accumulate in anticipation of environments and the lack of media affects the speed of their delivery. It occurs for two reasons. First, because of the lack of resources (in an attempt to do everything, people create a lot of unnecessary actions, and this slows down the overall process even more). And secondly, if out of a hundred changes a few dozen “fall” on tests, then this leads to an even greater shortage of environments, since after correction, you need to check everything again, and not just those who ruled. For example, changes in the configuration module can lead to a failure in the payment module and the failure of the entire application.
Thus, in order to increase the Agile effect, it is first necessary to implement two conditions:
1. The emergence of an opportunity to run business tests at no extra cost,
2. shift the integral and business testing as close as possible to the source of change.
And, of course, we need effective solutions for monitoring application performance in real conditions.
When will Agile's golden age come?
Since its inception, Agile has reincarnated from a set of principles into an assortment of methodologies, processes, and even standards. Today the field of activity of these methodologies is not limited to the development teams. Flexible processes are successfully implemented in almost all IT departments and even business is guided by Agile standards. Among the most famous are Scrum, Scrumban, SAFe, ScaleAgile @ Spotify, Continuous Delivery, Lean, Prince2 Agile and many others.
Despite the many frameworks and methodologies that, within Agile, claim leadership and declare that all problems are solved and all barriers are removed, quality remains the main gateway to unquestioning translation into Agile of all departments and development stages of IT corporations. The existence of this gateway is causing distrust of Agile and an attempt to justify the preservation of the principles of the cascade model or Waterfall. Attempts are also being made to change Agile by introducing the traditional stages of testing.
This will keep for a time the flow of errors flooding business processes, websites and corporate applications. However, in conditions of tough competition, business will not tolerate such a model for a long time. Only when translating the whole mechanism into flexible methodologies is it possible to achieve a business that is flexible and adapted to the requirements of the market, with high quality and speed parameters.
So, one of the key problems in the work of Agile today is the problem of quality. And its decision is a matter of the very near future. So, a number of tools have already appeared that help to quickly and accurately predict the degree of business satisfaction with new changes in applications. The introduction of such tools allows to achieve good results in the implementation of inspections.
Having solved this problem, in the next 5 years flexible methodologies will open their way to the remaining third of the corporations, which until then do not dare to translate the development into a more modern, but risky way.
Source: https://habr.com/ru/post/320782/
All Articles