Long road in the dunes, or how the ESF engineering subculture changed
When we published the first DevOps article in the ESF , we were asked why there was not a word about the process culture. Indeed, in the first place, DevOps is an IT partnership, a higher stage in the evolution of engineering culture, and only then - automation. In this article, we will describe how the outlook of employees was changed and what we discovered in this area.
As world classics accurately define, an organization is a complex organism, where the interests of an individual and groups, incentives and restrictions, rigid technology and innovations, unconditional discipline and free creativity, regulatory requirements and informal initiatives intertwine and get along. The younger the organization, the less well-established its values ​​will be. This makes it easier to change her culture.
The real way to change the engineering subculture was more difficult than we expected. It was necessary to interest developers with new technologies, instill responsibility for the entire process of developing their product, familiarize them with the headache of administrators and maintenance so that developers can climb into their territory without fear. And the developers in our team are more than 800, most are already rooted in the well-established values ​​of waterfall, which did not provide the right time-to-market for the purposes and projects of the ESF.
')
MIT professor Ed Scheine was right in that engineers and technicians are quite peculiar. Other subcultures seem to them incomprehensible and even somewhat fussy. Techmen focused on technical progress and the possibilities of technical creativity. Moreover, they rather want to survive people from their own world, rather than attract them there. Therefore, a simple way to implement DevOps through the establishment of KPI and automation of end-to-end production process did not suit us.
After studying the topic, we decided that we would try to change the culture from the inside. We will find evangelists "from the plow" in the development teams and we will show by their example that everything could be different.
We decided to start with a simple mailing list. They asked to respond to people who are not afraid of experiments, rakes and bumps.
From the recipient database at 800+ recipients, 5 people sent an application to participate in the pilot. Conversion of less than 1% of us was a little upset. Lev Gumilyov and his passionate theory of ethnogenesis were remembered: passionarity is an irresistible inner desire for activities aimed at changing one’s life, environment, status quo. According to the teachings of Gumilyov, Russia was a super-reliance with a high level of drive and the question is where did all go? What led our subpassionaries? Fear of change? Low involvement? Or did we just choose a bad communication method?
I remembered the summer conference on DevOps in Madrid, when many foreigners, learning that we were from Russia, got into Facebook and showed us their Russian fellow developers working in other countries. Has the majority of active passionaries from the IT environment migrated? And what do you think?
We have done a bit of passionarity a bit. So, our small, but enthusiastic team included experienced developers from Moscow and Innopolis (Tatarstan). Everyone had different experiences and goals, so the roles inside were distributed quickly. And the work began to boil.
A month later, the development team, together with the administrators and automation engineers, set up the first version of the pipe-line, started replicating to other commands and showed results. Instead of manually installing the console for 1.5 hours, it was carried out through a pipe-line for 10 minutes, and at night stability was checked. There was an automatic check of the lifting of the application context. In general, the optimization work did not stop, everyone wanted to make their product even better. Additionally, management introduced rewards for the work done.
The pilot team was not easy: the guys had to combine project tasks within their teams with the development of DevOps. Strong inner motivation for the result, overcoming process differences and communication barriers, helped.
Later, we derived our gradation of allied participants and the main approaches to working with them:
Those unfamiliar with technology are therefore afraid of the unknown. The result of communication with such a developer depends largely on its flexibility and willingness to change.
In teams, where there was a large proportion of young “neznaek”, the process of change was expectedly faster and easier. The main difficulty is regional distribution, so we conducted meetings and included colleagues online.
They know about technology, but they don’t want to change anything for a bunch of “valid” reasons: “not my area of ​​responsibility,” “I have been doing it for twenty years already,” “it's more important for me to write the code on time,” and so on.
It was more difficult with them, especially with those who openly opposed any arguments and examples. In this case, it was necessary to use administrative artillery, to hold joint meetings and conversations with the participation of the management team.
Which DevOps is better: kosher or orthodox? When we started the changes, the development of services was in full swing, some were already coming out in the prod. The ESF core development team managed to make its DevOps - as it could and with what it managed. Indeed, it was the best example of automation at that time, but the practice was only on dev-stands. It required major changes and modifications to scale, which upset many.
With this group, it was very interesting to implement changes, because it was based on very smart and experienced developers. We had vivid emotional discussions, which became much more productive after we conducted the first demo: when we showed how the pipe-line works and works, the colleagues themselves wanted to remake their own.
Unfortunately, there is such a category of developers who do not take responsibility for the quality of the software. For example, why do they write autotests? "Because KPI". Therefore, they write dummy tests to quickly get rid of this task. Consequences cover everyone, including storytellers: in emergency mode, you have to correct defects in your own and neighboring project, because the installation set by the auto-heating system broke the stand, and other teams are forced to rest while waiting for the stand to be restored.
This category turned out to be the most difficult for us. Well, that there were a few. You can work with them only through their own mistakes: they destroyed the stand because of the tests, blocked the work for the day for all the modules - debriefing.
In our work with objections, we often began a dialogue with the question: “Do you want not to spend any more energy on writing instructions?” It was a clue that really worked, no one likes to write instructions.
Of course, it wasn’t without wishes from the top, when the DevOps work was included in the backlog of the teams, and from each initiative a person responsible for setting up the pipe-line was selected. Participants were connected to the daily “stand-up” via corporate Skype - the EFS development team is distributed between Moscow, St. Petersburg, Rostov, Voronezh, Innopolis and other cities.
Unified communication and the tools that we use will be dedicated to a separate post. For work, we used Confluence, SharePoint, where we created a knowledge base with a sequence of actions for setting up Ansible scripts, laid out a communication matrix and a lot of useful information.
In addition to the stand-ups, once a week we conducted a demo on the work of the pipe-line. Over time, we organized the baton: the demo is conducted by the developer from the next team that set up the pipe-line. The process of incorporating teams into DevOps continues to this day; the approach of the relay allows you to always add something to the experience gained by predecessors.
Unfortunately, not all have reached the finish line of the pilot. Getting out of the comfort zone, the need to constantly prove the value of what you are doing, the double load and administrative delays (which without them) did not pass without loss in the team. Today, only 3 out of 5 ESU experimenters continue the development of DevOps, the rest have left the pilot and continued working on the current tasks in the teams. But these three are now connected to the new evangelists.
Could we avoid losses? Probably not. But if we were a little more careful, we would have seen the turning point before. We introduced the bonus motivation program for non-productive activity later.
As a result, we received evidence that process automation is not decisive. Success will be achieved when all 800+ participants will be responsible for their product to the end, and this will only be possible in the conditions of a particular engineering culture. It is difficult to assess the level of changes made, the choice of specific indicators is still ahead. And what do you think, what indicators DevOps will be indicative?
As world classics accurately define, an organization is a complex organism, where the interests of an individual and groups, incentives and restrictions, rigid technology and innovations, unconditional discipline and free creativity, regulatory requirements and informal initiatives intertwine and get along. The younger the organization, the less well-established its values ​​will be. This makes it easier to change her culture.
The real way to change the engineering subculture was more difficult than we expected. It was necessary to interest developers with new technologies, instill responsibility for the entire process of developing their product, familiarize them with the headache of administrators and maintenance so that developers can climb into their territory without fear. And the developers in our team are more than 800, most are already rooted in the well-established values ​​of waterfall, which did not provide the right time-to-market for the purposes and projects of the ESF.
')
MIT professor Ed Scheine was right in that engineers and technicians are quite peculiar. Other subcultures seem to them incomprehensible and even somewhat fussy. Techmen focused on technical progress and the possibilities of technical creativity. Moreover, they rather want to survive people from their own world, rather than attract them there. Therefore, a simple way to implement DevOps through the establishment of KPI and automation of end-to-end production process did not suit us.
After studying the topic, we decided that we would try to change the culture from the inside. We will find evangelists "from the plow" in the development teams and we will show by their example that everything could be different.
“Do not be afraid to make mistakes. To blunder sometimes is useful. Do not be afraid to break wood ”( S. Hanselman )
We decided to start with a simple mailing list. They asked to respond to people who are not afraid of experiments, rakes and bumps.
From the recipient database at 800+ recipients, 5 people sent an application to participate in the pilot. Conversion of less than 1% of us was a little upset. Lev Gumilyov and his passionate theory of ethnogenesis were remembered: passionarity is an irresistible inner desire for activities aimed at changing one’s life, environment, status quo. According to the teachings of Gumilyov, Russia was a super-reliance with a high level of drive and the question is where did all go? What led our subpassionaries? Fear of change? Low involvement? Or did we just choose a bad communication method?
I remembered the summer conference on DevOps in Madrid, when many foreigners, learning that we were from Russia, got into Facebook and showed us their Russian fellow developers working in other countries. Has the majority of active passionaries from the IT environment migrated? And what do you think?
“Even the journey of a thousand whether begins with the first step” (Lao Tzu)
We have done a bit of passionarity a bit. So, our small, but enthusiastic team included experienced developers from Moscow and Innopolis (Tatarstan). Everyone had different experiences and goals, so the roles inside were distributed quickly. And the work began to boil.
A month later, the development team, together with the administrators and automation engineers, set up the first version of the pipe-line, started replicating to other commands and showed results. Instead of manually installing the console for 1.5 hours, it was carried out through a pipe-line for 10 minutes, and at night stability was checked. There was an automatic check of the lifting of the application context. In general, the optimization work did not stop, everyone wanted to make their product even better. Additionally, management introduced rewards for the work done.
The pilot team was not easy: the guys had to combine project tasks within their teams with the development of DevOps. Strong inner motivation for the result, overcoming process differences and communication barriers, helped.
Later, we derived our gradation of allied participants and the main approaches to working with them:
- "Dunno" (more than 40%)
Those unfamiliar with technology are therefore afraid of the unknown. The result of communication with such a developer depends largely on its flexibility and willingness to change.
In teams, where there was a large proportion of young “neznaek”, the process of change was expectedly faster and easier. The main difficulty is regional distribution, so we conducted meetings and included colleagues online.
- Conservatives (30%)
They know about technology, but they don’t want to change anything for a bunch of “valid” reasons: “not my area of ​​responsibility,” “I have been doing it for twenty years already,” “it's more important for me to write the code on time,” and so on.
It was more difficult with them, especially with those who openly opposed any arguments and examples. In this case, it was necessary to use administrative artillery, to hold joint meetings and conversations with the participation of the management team.
- Religious fanatics (20%)
Which DevOps is better: kosher or orthodox? When we started the changes, the development of services was in full swing, some were already coming out in the prod. The ESF core development team managed to make its DevOps - as it could and with what it managed. Indeed, it was the best example of automation at that time, but the practice was only on dev-stands. It required major changes and modifications to scale, which upset many.
With this group, it was very interesting to implement changes, because it was based on very smart and experienced developers. We had vivid emotional discussions, which became much more productive after we conducted the first demo: when we showed how the pipe-line works and works, the colleagues themselves wanted to remake their own.
- Storytellers (less than 10%)
Unfortunately, there is such a category of developers who do not take responsibility for the quality of the software. For example, why do they write autotests? "Because KPI". Therefore, they write dummy tests to quickly get rid of this task. Consequences cover everyone, including storytellers: in emergency mode, you have to correct defects in your own and neighboring project, because the installation set by the auto-heating system broke the stand, and other teams are forced to rest while waiting for the stand to be restored.
This category turned out to be the most difficult for us. Well, that there were a few. You can work with them only through their own mistakes: they destroyed the stand because of the tests, blocked the work for the day for all the modules - debriefing.
In our work with objections, we often began a dialogue with the question: “Do you want not to spend any more energy on writing instructions?” It was a clue that really worked, no one likes to write instructions.
Of course, it wasn’t without wishes from the top, when the DevOps work was included in the backlog of the teams, and from each initiative a person responsible for setting up the pipe-line was selected. Participants were connected to the daily “stand-up” via corporate Skype - the EFS development team is distributed between Moscow, St. Petersburg, Rostov, Voronezh, Innopolis and other cities.
“Opportunities awaken reality, and nothing is more absurd than to deny it” (R. Musil)
Unified communication and the tools that we use will be dedicated to a separate post. For work, we used Confluence, SharePoint, where we created a knowledge base with a sequence of actions for setting up Ansible scripts, laid out a communication matrix and a lot of useful information.
In addition to the stand-ups, once a week we conducted a demo on the work of the pipe-line. Over time, we organized the baton: the demo is conducted by the developer from the next team that set up the pipe-line. The process of incorporating teams into DevOps continues to this day; the approach of the relay allows you to always add something to the experience gained by predecessors.
Unfortunately, not all have reached the finish line of the pilot. Getting out of the comfort zone, the need to constantly prove the value of what you are doing, the double load and administrative delays (which without them) did not pass without loss in the team. Today, only 3 out of 5 ESU experimenters continue the development of DevOps, the rest have left the pilot and continued working on the current tasks in the teams. But these three are now connected to the new evangelists.
Could we avoid losses? Probably not. But if we were a little more careful, we would have seen the turning point before. We introduced the bonus motivation program for non-productive activity later.
As a result, we received evidence that process automation is not decisive. Success will be achieved when all 800+ participants will be responsible for their product to the end, and this will only be possible in the conditions of a particular engineering culture. It is difficult to assess the level of changes made, the choice of specific indicators is still ahead. And what do you think, what indicators DevOps will be indicative?
Source: https://habr.com/ru/post/341608/
All Articles