Raiffeisenbank DevOps: Flight Phase

About DevOps does not tell only lazy. Some companies are introducing these practices, and the overwhelming majority is looking closely at the next big thing or the “silver bullet”, or simply succumbing to the trends in the IT community. The uniqueness of each case, the search for your own path, the fear of making things worse (the Hippocratic principle “do no harm”) - all this does not accelerate the implementation, just adding steps to the perfection of IT. We want to tell you about your path, winding and not yet passed to the end.



In the beginning there was pain. There were a lot of problem areas that bothered different teams, and the problems in the existing paradigm of administrative division into “friend or foe” clans (developer - support) did not work out. The speed of implementation of changes was none - the installation of an important patch could take more than one day, let alone big releases ... For testing, it was necessary to put the change on a test environment that was managed by a centralized service, but there was always a line for it, and there were several to wait days Level 44 experts spent their precious time manually installing updates. Rollback updates were unpredictable in time. An initiative not invested in the form of a legal registered project could be easily buried, despite the initiator’s charisma - in adjacent departments you could be denied any initiative or put it on such a pause, after which it was pointless to return to the idea - the market “left”. On the other hand, how to exist in a large organization without regulations, procedures, approvals, all that was created to organize the chaos, and turned into a bureaucracy?

In this coordinate system, we set out to reduce time to market. At the time, the DevOps theme was a hot topic on the market. As it turned out, the story is not only about automation, but about relationships in teams. We decided to try it on ourselves. The heads of the development, support and infrastructure departments took up the development and promotion of a new ideology. At the nearest IT conference Rife, the leaders said loud words, and there was no way back - they “signed up” for the system-forming changes. It would seem that "the goals are outlined, the plans are defined, for the work, comrades" - but everything went wrong. After the words Myths arose ... As at the dawn of mankind, what was happening around filled with meanings, fear filled the minds, and events were perceived through the prism of someone else's experience. In the smoking room you could hear all 50 reasons to do nothing - and it doesn’t matter if you have an old school cigarette in your hands or a vape and a beard .
')
In the end, the mythology around DevOps was studied and framed as a collection.

DevOps Myths Raiffeisenbank
	We have already launched DevOps
	DevOps is not for serious companies
	Developers will understand the infrastructure, and engineers will code
	ITOps teams will not be needed, because everything will move to the clouds
	The main thing when implementing DevOps is to choose the right toolkit and nothing else is needed
	DevOps for the developer: now everything is possible!

Every manager who was afraid to sleep without light was met and explained that there are no particular reasons for fear, and the job security provision criterion, as always, is an excellent working result. In the meantime, that is, spontaneously and a little earlier :), the initiative arose and began to develop by the employees themselves - the most involved, those who care. A stable product with frequent deliveries of releases, and the delivery itself with a high frequency in the current conditions was difficult to obtain. It was necessary to change the approach to the task, determine the rules of interaction and roles, expand the boundaries of responsibility - without fanaticism, of course.

As a result, two DevOps models have been formed, which the teams today adhere to. They featured new roles: Support Expert and DevSupport expert, which develop and exploit a set of team-based practices aimed at more detailed immersion of colleagues in each other's processes. New agreements are being arranged, the main problems of interaction are being solved, expectations are being expressed from both sides.

Shared DevOps "The overall goal is both speed and reliability"

Embedded DevOps “All team members are product focused.”

This model is the primary and necessary in all teams. Dev + Ops has one employee on each side (Support Expert + DevSupport Expert). The model assumes a team focus on their goals. Team members are maximally involved in processes (Run + Change). Everyone understands that he is working on a common product and is solely responsible for its stability and development.

This model implies the presence of a DevOps team, which is generally responsible for the development and support of the E2E product, each member who can perform different roles. Support Expert is a member of the SCRUM team and is fully allocated to all activities of this team. Such a model is possible when the team develops a fairly stable product and the amount of operational work is low, as well as if there is such a need on the part of the product owner.

The stories about the results achieved at the internal meetings, word of mouth and professional pride did their job - very quickly all the teams began to try to automate testing, delivery, assembly.

At first there were relatively “simple” stories with “correct” technologies - they coped with them themselves, but on some boxed monsters they had to “turn on” external vendors. In all the episodes, an enormous role was played by the internal center of competence and the enthusiasm of some colleagues: thanks to them, they were able to roll out CI / CD even on dinosaurs, which are not inherently suitable for automation.

There were unspoken standards: something that has proven itself to be an effective solution and has been agreed in our infrastructure. Such solutions could be rolled out in your team very quickly, without being associated with approvals, budgets, procurement, and most importantly - already having an internal expertise. What started here ... At every internal meeting in IT, in each speech, there were words about DevOps, about achievements (no one likes to talk about failures), the development of the metrics of the degree of “devotion” of the team began. Those of us who supported the products with releases every six months reflexed and justified: they say, I would like to introduce, but there is no special meaning, the labor costs do not pay off. The scale of the movement and the variety of ways and practices required some leveling - both standards and knowledge.

For several years now, our large team has had an internal training program - IT Academy. The set of courses, and indeed the teaching method itself, has experienced a number of transformations over several years, as a result of the Academy, in terms of the approach to education, only a pathetic name remains. The essence of successive changes - the transition from pumping theoretical knowledge to practical work. We tried to hire cool trainers, drive all teams through trainings, distribute certificates and get nothing substantial, no working solution at the output. The restart of the Academy was very pragmatic: the teams got a lot of problems at their disposal interesting, complex tasks, resources in the form of consultations of internal and external experts, virtual platforms for testing hypotheses and experiments. By the end of the “semester” of the task, for example, automatic delivery to Production, should be solved. There was a queue of those who wanted to go through such an Academy. Training on their own problems, as they say, “has gone”: the vital need to introduce solutions and knowledge, integrate them into their work, created the necessary motivation and there were practically no contributions from the Academy. Progress is noticeable and measurable - after all, we also implemented the DevOps metrics and accumulated some statistics.

Why so much pride in telling about some successes, and, in general, failures in the way of implementing DevOps? In short - because the enterprise. In the following publications I really want to reveal this topic: systemic, large-scale changes in large (by the standards of IT) teams. We plan to tell you about the culture, the one that eats breakfast strategy. We want to talk about the technical details of automation solutions: autotest, CI / CD, automatic creation and configuration of infrastructure, because without this DevOps will be just an inspiring story. It took a lot of time to develop an approach to DevOps metrics - and it is definitely worth telling about this too. Finally, it is interesting to find out how the distance traveled looks through the eyes of a developer, system administrator, technical support expert — for sure in different ways. In general, the topic is developing, follow the publications, leave comments, which topic is more interesting for you, and we will tell about it first.