Sharing Scrum and DevOps - Translation of the article The Convergence of Scrum and DevOps

Translation of an article written by Scrum.org and the DevOps Institute. Link to the original file

From translator

The article seemed to me very useful, although difficult to translate, despite the fact that some of the terms that refer to Agile are known to me. I tried very hard not to distort the meaning of the original and I hope that I succeeded. In any case, anyone who speaks English, I strongly advise you to read the original. This is my first job in the field of public translation, so please do not judge strictly.

In essence, an article is practically a user guide (albeit at an extremely high level). The only thing is that the advice from it cannot simply be taken and implemented (which probably applies to any methodology), and, from my point of view, the main principle should be adhered to - the changes should be smooth and should not break what works. Any changes should flow out of pain (large or small), then the team is ready for them, no need to create this pain artificially.

I put the links in the main document right away into the text, they are separated by brackets and italics. If there were doubts about the correctness of the translation of the term, I duplicated it in brackets in English.

Sharing Scrum and DevOps

"Software absorbs the world"

(Andreessen, Marc. "Why the Software Is Eating the World." The Wall Street Journal. August 20, 2011.
https://www.wsj.com/articles/SB10001424053111903480904576512250915629460 ).

Software (software) is everywhere, it affects all aspects of modern life. It allows organizations to create more advanced products, as well as better understand customer needs.

Software is also a source of change. Those organizations that take full advantage of automation bypass their "non-digital" competitors ("2017 State of DevOps Report." Puppet. Https://puppet.com/resources/whitepaper/state-of-devops-report ) due to that hypotheses are tested faster, the results of experiments are obtained and analyzed, and this information is used to improve.

Accelerated by new technologies, markets and climate change, the world is changing faster and faster (FRIEDMAN, THOMAS L. THANK YOU FOR BEING LATE: an optimistic guide. Thirving in the age of accelerations. Sl: PICADOR, 2017) . The only way to survive is to take the path of change.

Two methodologies - one goal

Agile and DevOps are the main ways in which organizations achieve the same goals: faster development cycles, fewer functional blocks for a release, using feedback to improve processes, removing obstacles and untargeted effort. The aspects that are emphasized in these two methodologies are different, because sometimes people think that these are different approaches. Agile focuses on teamwork, culture and values, while DevOps puts the development pipeline (delivery pipeline) and its continuity (flow) at the head. But Agile also concerns the topic of automation, and DevOps - communication and culture.

DevOps and Agile should be used simultaneously. One of the most popular Agile development methods is Scrum, which is used daily by up to 18 million IT professionals ( https://www.infoq.com/news/2014/01/IDC-software-developers ) , which makes it (and Agile in general, a key component of the development, that is, at a minimum, provides the “Dev” part of the term “DevOps” (translator's note: Dev - from English developement - development, Ops - operations - operation).

Both methodologies seek to create value for the user in the most efficient way, but their initial focus is different. In Agile, it's a developer, and in DevOps, it's a process. These differences are reinforced by:

• Some Agile supporters think that any process is bad, and "Ops" has a rich history of ensuring stability through building a process.
• In DevOps, instructions (tools) and automation are used to increase speed, while those who use Agile, at best, see instructions as an inevitable evil, and at worst, a hindrance.
• The DevOps concept addresses systems thinking to solve problems, while Agile encourages people to be drivers of change.
• Political games and a lack of communication between business units intensify the gap.

Continuing change requires strong leaders. They provide not only clarity of purpose and direction of development, but also authorize and stimulate change. Changes in organizations also attract strong personalities who earn their credibility as their driving force and are rewarded when changes produce results.
When the success of these people is associated with a program of change, it may be a problem that different programs of action will start competing among themselves. When the implementation and use of Agile and DevOps are promoted by different parts of the organization, competing policy goals lead to confusion:

• Hostile to each other stakeholders . Business partners and managers see conflict and lose confidence that at least someone understands what they are doing.
• The clash of cultures between development and exploitation . The open, people-centered community of Agile is very different from the process-oriented exploitation group.
• Growing fear and defensive behavior . Nobody wants to increase the level of risk, but at the same time it works with it differently. In Agile, risks are reduced by numerous, but few risk changes. From the point of view of Ops, a large number of changes are inherently risky, because the response is strengthening orderliness, control and management.
• Developers view DevOps as a temporary measure; they see the target as “NoOps”. NoOps means full automation of operation, allowing developers to manage the production environment. Whether this is realistically possible or only desirable is generally not important; most organizations are still so far from effectively using the DevOps concept that NoOps is a vague and useless goal.

The reality is that the value creation process, which is transmitted to customers through the provision of software, includes as one type of element, which requires a detailed, comprehensive process and another, which is flexible and requires the team to work in an energetic manner. Automation is a process that ultimately stabilizes everything, but for many situations automation is impossible, or its cost outweighs the possible benefits. The balance between automation, process and teams with a lot of authority is important. If you focus on one and ignore the others, this leads to:

• Low quality. A systematic approach to quality is crucial for the quality of any product, but for many teams, quality is still provided by a small-town, at best, manual testing that surrounds their continuous integration process.
• Nothing is ever completed . If there are too many complex processes that require external coordination or verification, it becomes almost impossible for a team to bring the backlog task to the “Completed” state.
• Lack of transparency and visibility . Many processes are so complex that no one can cover them entirely, and it becomes impossible to make informed decisions.
• With changes in the composition of the team changes everything . Without a clear and understandable process, it is difficult to manage changes related to people or to scale something.
• The appearance of hidden work, which the process does not include explicitly . As the processes become more detailed and comprehensive, more and more work arises, which is not included in them (translator's note: I understood it this way: more detailed processes take more and more time, but no process can cover everything, therefore works that are not included in them, which also need to be done). This leads either to a large amount of “hidden” work and work “bypassing” or to work in vain.

Combining Scrum and DevOps.

Imagine a world where cross-functional, business-oriented teams constantly release working software and where workflow flows seamlessly between all relevant stakeholders in real time. Add to this that value is clearly recognized, measured and reported on. This vision is a reality for many small startups that have mixed business and technology and work with customers and the market to bring customers value. Scrum teams are responsible for developing software, and operating teams are responsible for maintaining an environment that allows you to do this safely and efficiently.

Cross-functional teams with a set of necessary skills deliver the software developed to the end faster.

Creating cross-functional teams is not new. Hirotaka Takeuchi and Ikujiro Nonaka described the importance of flexible, cross-functional teams in The New New Product Delivery Game in 1986 (Nonaka, Hirotaka TakeuchiIkujiro. "The New New Product Development Game." Harvard Business Review. August 01, 2014. https://hbr.org/1986/01/the-new-new-product-development-game ). But to make the right decision about who should be included or not included in the team is difficult. Modern products are a stunning array of different technologies that historically required special skills and infrastructure. And the development team must have all the skills necessary to create a work product. It is described in the Scrum Guide (The Scrumuide. Http://www.scrumguides.org ) as:

"Cross-functional teams have all the competencies necessary to perform the work and do not depend on those that are not part of the team."

Cross-functional teams must have the necessary skill set to complete the work. When a team transitions from a traditional role-based approach, its members should increase the number of skills they have. They also need to learn to feel sympathy (build empathy) for the customer in order to make better products. This does not mean that every single person can be only in one team. There are a large number of examples where someone with expertise in operation is a member of several teams at once (acting almost as a “service unit” for a team), bringing their experience and knowledge to the team (one or several), which allows to improve the product development process, along with training other team members so that they can expand their skills. But this should be in balance with how their availability affects the ability of the team to develop tasks from backlog. They also, as a rule, should:

• Reduce the number of tasks to be transferred (hand-offs). Reversing tasks increases wait time and reduces code ownership and responsibility. Flipping tasks may be inevitable when a team lacks a full set of necessary skills or authority to develop a complete solution. But if this concerns most of the work that is done by the team, then people with the necessary skills should be included in the team.
• Develop all the skills you need to create value. When teams do not have a full set of necessary skills, they will need the help of specialists. Over time, they should strive to learn from these specialists in order to increase their own independence. For example, at the beginning of the development of any application, the issue of security should be taken into account. Those who are responsible for security can help the team, the team can learn from them, reducing their dependence and improving the quality of work.
• Increase process automation. Automating a large number of tasks that a team performs mechanically in the software development process frees up time for valuable work focused on customer satisfaction. These mechanical tasks include at a minimum: continuous integration, testing, deployment and preparation of the environment.

DevOps and Scrum are used successfully when obstacles are removed.

Queues reduce the team's flexibility, as they deprive the team of the ability to control the situation. When support communicates with teams working on Agile through queues, it is very likely that the team as a whole will be less effective, as it will wait until its request reaches the beginning of the queue.

When building a future development system, you should eliminate the queues step by step and replace them with self-service capabilities. Such traditional functions that relate to IT, such as: maintaining the environment, databases, security, network infrastructure, as well as software deployment, should be supported as a set of self-service systems. These systems will be very similar to services that companies such as Amazon Web Services (AWS), Microsoft Azure and Google currently provide and will often use these external cloud services, adding organization-specific intellectual rights to them to make them compliant with clomplience- company policy (company compliant).

Legacy applications that have long existed in a company can also reduce flexibility, since over time they become so complex that no one understands how they work. The complexity of these applications makes it difficult to achieve all the necessary skills in one team so that it can do tasks independently. For example, the amount of work and skills required to create a test that fully covers the integration process makes it almost impossible to complete this task in a single sprint! Unfortunately, there is no silver bullet for a quick solution to this issue. To do this, the company must rewrite the architecture of this application to enable modular development, deployment and testing.

Helping teams increase their abilities includes:

• Teams must have the authority to perform all that is required to release a product. Instead of imposing a large number of restrictions, in order to protect the team from itself, one should give it the authority to make decisions and develop the product entirely. But it should be ensured that, firstly, the scale of the expansion of powers was no more than the one that allows you to easily roll back the changes, and secondly, the development environment and the applied architectural principles were sufficiently stable to withstand the changes.
• Reduction of external dependencies, through automation, built on self-service automation (self-service automation). Instead of including Agile commands in complex processes, provide them with self-service services that allow Scrum teams to produce results autonomously. For example, allocating virtual machines or providing access to test data.
• Ensuring that teams have access to technical resources that are available without waiting in a queue. The simpler the software, the more flexible the team can be in its development and support. With the growth of its complexity and the importance of a single team, it becomes harder and harder to have all the necessary skills to support it. But any dependence on external resources can become a bottleneck. Concentrating on the support of the Scrum teams by experts who can train and support the team in using a certain technology, you simultaneously eliminate the bottleneck and improve the team skills, making it flexible and Agile.
• Simplification and module breaking of old (legacy) applications. Strive to recycle old systems and support infrastructure such as development environments and tests to better maintain agility. This means that you need to focus on reducing the limitations in testing, providing a better division into modules and spreading knowledge about old systems across all teams, rather than concentrating it in one place.

Automation, automation and automation again

Continuous software delivery is one of the most important DevOps practice sets. In its best implementation, it looks like this: every time the developer commits the code, it is included in the build (gets build), tested, deployed (if it meets all the release criteria) with little or no human intervention. The lack of manual labor not only speeds up releases, but also improves its quality by removing the probability of error due to the human factor.

The first step of the team in the process automation process is its analysis for the presence of work that does not carry additional value. At the same time, the application architecture is being revised. As a result, a new backlog of tasks may appear, the purpose of which is to recycle the application and eliminate technical debt. Here are some good ideas for automating:

• Simple processes are best. Get rid of over complicated processes (complexity and over-engineered processes).
• Start from the beginning and from the end of the process and move to its middle. Unit tests and deployment to the production environment are perhaps the most appropriate areas to start writing scripts.
• Automation should be a natural part of the work of Scrum teams. You might think that there should be separate teams that are engaged in automation and build a standardized environment. Experience shows that these teams, which support centralized tools, strive to work for everyone, but as a result do not provide services to anyone in full. Focus Scrum teams to do the right thing to meet their own needs. These teams must work together with the operation to be sure that everything that is being done can be used both in software deployment and in the production environment.

Software development professionals

At the heart of the modern organization that develops software are qualified professionals who can apply their experience (apply an empirical process) and who have the necessary engineering / operational skills to ensure that their work is at the right level of quality.

Scrum believes that professionalism is so important that Scrum-Guide includes five
values: concentration, openness, respect, commitment and courage.

These values ​​can help organizations take a holistic look at their own ability to develop software by adding a new metric to the development process. There are several real ways that professionalism can be encouraged in a modern development organization:

• Enter an apprenticeship, mentoring, example training in personal development practices. Based on medieval methods of knowledge transfer, this approach to building a career creates a community of students and teachers in accordance with the principles of lean thinking, whose leaders leave the meeting room and office to work with a team in Gembla or elsewhere value is created.
• Reward teams, not individuals. In most organizations there are few areas where a person can achieve something alone. A large number of different people should work together to bring more benefits to the consumer. Most initiatives that are tied to one person are irrelevant, and personal rewards are contrary to the fact that the result is very rarely achieved alone.
• Reward people for real achievements, and not just for being busy with something. Focus on increasing customer value and eliminating unnecessary work, rather than keeping people as busy as possible. As long as human resources are expensive, no one gains from the fact that busy people give the wrong decision. And the maintenance of constantly busy people whose skills are scarce simply means that many others will have to wait.
• Make these values ​​as important as the result of the work . How often did you hear someone say, “Yes, he managed to complete the development, but it was a bloodbath”? Such a heroic approach to development sometimes works, but its result is unpredictable and destroys the organization, leading to burnout of people and creating huge amounts of technical debt. Contributing to the fact that professionals live according to the values ​​that Scrum is focused on (An excellent overview of Scrum values ​​can be found here: "There's value in the Scrum Values." Ullizee. May 03, 2013. https://guntherverheyen.com/2013/05 / 03 / theres-value-in-the-scrum-values ​​/ ) , openness, respect, commitment and courage, you do not just create an environment that promotes flexibility, but in which you also want to work. If development, operation, and business teams also share these values, mutual trust will appear by itself. In the course of the account, concentrating on the common values ​​for the organization, you build quality products, not only in terms of the absence of defects, but also in terms of value to the user.

Integrate, concentrate and win

For many, the barriers that prevent becoming holistic, having a well-developed internal collaboration, cross-functional organization engaged in development and support are insurmountable: the status quo seems unbreakable, legacy applications and culture are immutable. Attempts by some organizations to change result only in the assignment of old processes, roles, organizational structures of new names.

But the pragmatic, step-by-step process of moving towards Agile and DevOps can help find a way forward, with some positive changes becoming noticeable in the process. Not all parts of the business require the increased flexibility that Scrum and DevOps give. Thus, gradually select those parts of the business that will be positively affected by the new methodologies, and change them.

Changing an organization sounds like a difficult task, and it would be if all changes needed to be made at once.

Changes begin with a customer who could be better served and with a developer organization that wants to make it more satisfied.

Step 1: Unification of the approach to software delivery

This may seem simple, but many organizations will find that they follow an approach that divides the process between groups of professionals who are focused on reducing costs and risk, rather than seeking to bring benefits to the consumer. Work flows from the planning function to the development, testing, output to production, and then operation. Development and operation teams are separate, and within projects, a situation may arise where different groups of people work on the same system and business area.

Thus, the first step encourages the organization to stop this insanity by adopting a holistic, product-centric approach to creating business value. Projects may continue to exist, but they are secondary to stable teams, including operation, which adds new value by meeting the need to keep the environment in working condition. Large products that are developed by teams can use the Nexus framework ("Scaling Scrum with Nexus." Scrum.org. Https://www.scrum.org/resources/scaling-scrum ) to scale Scrum to create Scrum- teams, while small products can be developed using Scrum.

Here are the conclusions for the Scrum team:

• Creating teams that focus on the product and customers. Focusing on a product or customer means that the product backlog will contain both new tasks and support tasks. If the product is large, then these teams will be organized around Nexus, which simplifies the work of a large number of teams on one backlog of the product.
• The inclusion of suitable people in order to deliver a fully finished product. And when we say a finished product, this means that it is put into a production environment and transferred to support. To fully comply with the “Difinition of done” criteria for software (“Definition of“ Done ”.” The Scrum Guide. Http://scrumguides.org/scrum-guide.html#artifact-transparencydone ) requires the involvement of all the necessary people. Operation, security, data management will either be included in the teams, or they will empower Scrum teams to take on tasks that traditionally fall within their area of ​​responsibility. This requires changes from organizations, meaning a departure from a single job and a task, being part of a team, helping others do the job and ensuring it is correct.
• Responsibility for the full release cycle, not just for the development part. The fundamental change for Scrum teams that still live in the Water-Scrum-Fall paradigm, which is necessary for them to be effective and bring real business benefits, is that they must become fully responsible. for the complete process of creating value as a team, breaking the barriers of possessiveness and individualism.

And such - for the service operation:

• Restructuring supported services to group customers / products. The first step is the desire to work with applications in relation to the business, and restructuring based on business results and products, rather than based on the processes and skills of specialists.
• Remove queues from the system. When a Scrum team cannot do the work on its own, do not make it wait for someone else to get help; make sure that help is available at any time when the Srcum team needs it. , , "" , , , . .
• Scrum-. , , Scrum-. Scrum', - Scrum-, , . , , , , .
• , . Scrum- — , , . — , (holistic) .

2. .

, , , , , , , , -, . lean/- . , , .

Scrum- :

• . (Sprint Review) — , ; , , . , . Nexus , , , , " ", , .
• . , , , . (Emperical) . , , .
• . , , . Scrum- (stakeholders), . — , , Scrum- ( (Stakeholders)), , ( (release), - , ). , , .
• () , . , .

:

• . , , , . , , Scrum- .
• (dashboards), . , , . , , , , .
• -, Scrum', . , Scrum- (review). Scrum — ; , , , . Scrum-, , .
• , , , , DevOps. Scrum' , . — , , . . (release management), . () .

3. .

Scrum- , Scrum- , .

(This might sound like semantics), , Scrum-, , — , , Scrum- . Scrum- , , . "" . Scrum- , , , . :

• , (evidence-based measures), . ( : (evidence-baced) ) — , : , ("Evidence Based Management (EBMgt)." Scrum.org. https://www.scrum.org/resources/evidence-basedmanagement ). , .
• . . , . , — , . / — , - , , , .
• "" . ; , , , , , . Scrum- /, . , , , .
• , (stakeholders). , , , .

Conclusion

DevOps , Agile- "Agile-". Agile- . , , , IT. , DevOps . Scrum/Agile- . , , . DevOps Scrum-, , , , (by inspection and adaptation through transparency).

, Scrum-. . — , . , Nexus, , , Nexus (Nexus Integration Team (NIT)), Scrum-. , , /, . , , — , , Agile. , , :

• - ? , , Scrum- , (agility) .
• , . , , Scrum-, — Scrum-. , .

. DevOps / . , . , . , , , . , :

• . , . , , , , , . , , . , , , , .
• , , . , , , , . , - , .
• , , . , . DevOps (make them part of the solution).

, , , , . , , , , . , . , , , , . , , Agile- — , "Agile" , (the opposite of agility). , (empirical), . .