DevOps LEGO: how we layered pipelines
Somehow we put the electronic document management system to the customer on one object. And then to another object. And one more. And on the fourth and fifth. We got so carried away that we reached 10 distributed objects. Powerfully it turned out ... especially when we reached the delivery of changes. As part of the delivery to the production loop for 5 test system scenarios, it took 10 hours and 6-7 employees. These costs have forced us to deliver as rarely as possible. After three years of operation, we could not stand it and decided to spice up the project with a pinch of DevOps.
Now all testing takes place in 3 hours, and 3 people participate in it: an engineer and two testers. Improvements are clearly expressed in numbers and lead to a reduction in the beloved TTM. In our experience, there are far more customers that DevOps can help, than those who know about it at all. Therefore, in order to make DevOps closer to people, we have developed a simple constructor, which we will discuss in more detail in this post.
Now tell more. In one energy company, a technical document management system is being deployed at 10 large sites. In projects of this scale without DevOps to swim out is not easy, because a large proportion of manual labor greatly delays the work and also reduces the quality - all manual work is fraught with errors. On the other hand, there are projects where the installation is one, but it is necessary that everything works automatically, constantly and reliably - for example, the same workflow systems in large monolithic organizations. Otherwise, someone will make the setting manually, forget about the deployment instructions - and as a result, the setting will be lost on the prode and everything will collapse.
')
Usually we work with the customer through a contract, and in this case our interests diverge a little. The customer looks at the project strictly within the budget and TK. It can be difficult to explain to him the benefits of various DevOps practices that are not part of the TK. And if he is interested in quick releases with added value for the business, in building an automation pipeline?
Alas, in work with a pre-approved cost, this interest is not always possible to grope. In our practice, there was a case when we had to pick up the development of an unscrupulous and sloppy contractor. It was a horror: there are no actual source codes, the code base of the same system is different for different installations, the documentation was partially missing, and partly was of terrible quality. Of course, the customer did not have control of the source code, build, releases, etc.
So far, not everyone knows about DevOps, but it is worth talking about its advantages, about the real saving of resources, eyes light up at all customers. So the requests including DevOps, over time becomes more and more. Here, in order to easily speak the same language with customers, we need to quickly connect business problems and DevOps practices that will help build a suitable development pipeline.
So, we have a set of problems on the one hand, there are knowledge, practices and tools of DevOps on the other. Why not share your experience with everyone?
Agile has its manifesto. ITIL has its own methodology. DevOps was less fortunate - he hasn’t yet grown over with templates and standards. Although some try to determine the degree of maturity of companies, based on an assessment of their methods of development and operation.
Fortunately, in 2014, the well-known company Gartner gathered and analyzed key DevOps practices and the relationships between them. Based on this released infographics:
We took it as a basis for our designer . In each of the four spheres there is a set of tools - we collected them into a database, selected the most popular ones, identified integration points and suitable optimization mechanisms. In total, 36 practices and 115 tools turned out, a quarter of which is open source or free software. Next, we will tell you about what we have done in each area and, for example, about how this was implemented in the technical document management project from which we started the post.
In the notorious project on the EDS, the technical documentation management system was deployed along the same scheme on each of 10 objects. The installation includes 4 servers: a database server, an application server, full-text indexing and content management. In the installation they work within a single node, are placed in the data center at the sites. All objects differ slightly in infrastructure, but this does not interfere with global interaction.
At first, according to the DevOps practices, we automated the infrastructures locally, then we brought the supply to the test loop, and then to the customer’s production. Each process worked step by step. The settings of the environments are fixed in the system of source codes, with which the distribution kit is assembled for automatic updating. In case of changes in the configuration, the engineers simply need to make the appropriate changes to the version control system - and then the automatic update will take place without problems.
Thanks to this approach, the testing process is greatly simplified. Earlier in the project were testers, who only did what they manually updated the stands. Now they just come, they look, that everything has earned and are engaged in more useful things. Each update is tested automatically - from the surface level to the automation of the business scenario. The results are laid out in the form of individual reports in TestRail.
Continuous experimentation is best explained with a test design example. To test the system, which is not yet - creative work. When writing a test plan, you need to understand how to test correctly, which branches to go through. And find a balance between time and budget to determine the optimal number of checks. It is important to choose exactly the necessary tests, consider how the user will interact with the system, take into account the environment and possible external factors. Without continuous experimentation is indispensable.
Now about the culture of interaction. There used to be two opposing sides - engineers and developers. The developers said: “We don’t care how it runs. You are engineers, you are smart, make it so that it can be operated without failures . ” The engineers responded: “You, the developers, are too careless. Let us be careful, and we will put your releases less often. Because every time you put a code through to us, it’s not clear how we interact . ” This is a problem of cultural interaction, which from the point of view of DevOps is built differently. Here you and engineers, and developers are part of a unified team that is aimed at the ever-changing, but at the same time reliable software.
On the scale of one team, specialists are set up to help each other. As it was before? For example, some thick deployment instructions were prepared, for 50 pages. The engineer read it, did not understand something, swore and asked the developer to comment at three in the morning. The developer commented and also cursed - in the end, no one was happy. Plus, naturally there were some mistakes, because I don’t remember everything in the instructions. And now the engineer, together with the developer, is writing a script for the automated deployment of the application software infrastructure. And they talk to each other in almost the same language.
The size of the team is determined by the scale of the update. The team is recruited during the formation of the supply, it includes people from the general project team. Then an update plan is written with those responsible for each stage, as the team executes it reports. All team members are interchangeable. As part of the team, we also have a safety net developer, but he almost never has to connect.
There are few points in the technology scheme, but under them are a lot of technologies - with their descriptions you can release a whole book. So we will highlight the most interesting.
Now, probably, this concept will not surprise anyone, but earlier the descriptions of infrastructures left much to be desired. Engineers looked at the instructions with horror , the test environments were unique, they were holili and cherished, and dust particles were blown away from them.
Now no one is afraid to experiment. There are basic images of virtual machines, there are ready-made scenarios for deploying environments. All templates and scripts are stored in the version control system and updated promptly. Previously, when it was necessary to deliver a package to the stand, a configuration gap appeared. Now you just need to add a line in the source code.
In addition to infrastructure scripts and pipelines, the Documentation as a Code approach is also used for documentation. Thanks to this, it is easy to connect new people to the project, acquaint them with the system by the functions described, for example, in a test plan, and also reuse test cases.
In the last article about DevOps, we explained how we chose tools for implementing continuous delivery and monitoring. Often, you do not need to rewrite anything - it is enough to use previously written scripts, build the integration between the components correctly and make a common management console. And all the processes can be started by a single button or schedule.
There are different concepts in English, Continuous Delivery and Continuous Deployment. Both can be translated as “continuous delivery”, but in fact there is a slight difference between them. In our project for the technical documentation of the distributed energy company, rather, the Delivery is used - when the installation for the prod goes on command. In Deployment, the installation is automatic. Continuous Delivery in this project has become a central part of DevOps .
In general, using the collection of certain parameters, you can clearly understand the usefulness of DevOps practices. And bring it to the leadership, which is very fond of numbers. The total number of launches, the execution time of the stages of the script, the proportion of successful launches - all this directly affects everyone’s favorite time to market, that is, the time from committing to the version control system to releasing a version on a production environment. With the introduction of the necessary tools, engineers receive valuable indicators by mail, and the project manager sees them on a dashboard. So you can immediately appreciate the benefits of new tools. And you can try them on to your infrastructure with the help of a DevOps constructor.
We will not prevaricate: for a start it became useful to us. As we have said, the customer needs to speak the same language, and with the help of the DevOps designer, we can quickly sketch out the basis for such a conversation. Specialists from the business will be able to assess for themselves what they need, and thus develop more quickly. We tried to make the designer as detailed as possible, added a bunch of descriptions so that any user would understand what he chooses.
The format of the designer allows you to take into account the existing developments in the company on building processes, automation. No need to break everything and build again, if you can only choose solutions that are normally integrated into existing processes that can simply fill in the blanks.
Perhaps your development has already moved to a higher level and our tool will seem too "captain". But we find it useful for ourselves and we hope that it will be useful to any of the readers. We remind the link to the designer - if anything, you get the scheme immediately after entering the source data. We will be grateful for the feedback and additions.
Now all testing takes place in 3 hours, and 3 people participate in it: an engineer and two testers. Improvements are clearly expressed in numbers and lead to a reduction in the beloved TTM. In our experience, there are far more customers that DevOps can help, than those who know about it at all. Therefore, in order to make DevOps closer to people, we have developed a simple constructor, which we will discuss in more detail in this post.
Now tell more. In one energy company, a technical document management system is being deployed at 10 large sites. In projects of this scale without DevOps to swim out is not easy, because a large proportion of manual labor greatly delays the work and also reduces the quality - all manual work is fraught with errors. On the other hand, there are projects where the installation is one, but it is necessary that everything works automatically, constantly and reliably - for example, the same workflow systems in large monolithic organizations. Otherwise, someone will make the setting manually, forget about the deployment instructions - and as a result, the setting will be lost on the prode and everything will collapse.
')
Usually we work with the customer through a contract, and in this case our interests diverge a little. The customer looks at the project strictly within the budget and TK. It can be difficult to explain to him the benefits of various DevOps practices that are not part of the TK. And if he is interested in quick releases with added value for the business, in building an automation pipeline?
Alas, in work with a pre-approved cost, this interest is not always possible to grope. In our practice, there was a case when we had to pick up the development of an unscrupulous and sloppy contractor. It was a horror: there are no actual source codes, the code base of the same system is different for different installations, the documentation was partially missing, and partly was of terrible quality. Of course, the customer did not have control of the source code, build, releases, etc.
So far, not everyone knows about DevOps, but it is worth talking about its advantages, about the real saving of resources, eyes light up at all customers. So the requests including DevOps, over time becomes more and more. Here, in order to easily speak the same language with customers, we need to quickly connect business problems and DevOps practices that will help build a suitable development pipeline.
So, we have a set of problems on the one hand, there are knowledge, practices and tools of DevOps on the other. Why not share your experience with everyone?
Create a DevOps Designer
Agile has its manifesto. ITIL has its own methodology. DevOps was less fortunate - he hasn’t yet grown over with templates and standards. Although some try to determine the degree of maturity of companies, based on an assessment of their methods of development and operation.
Fortunately, in 2014, the well-known company Gartner gathered and analyzed key DevOps practices and the relationships between them. Based on this released infographics:
We took it as a basis for our designer . In each of the four spheres there is a set of tools - we collected them into a database, selected the most popular ones, identified integration points and suitable optimization mechanisms. In total, 36 practices and 115 tools turned out, a quarter of which is open source or free software. Next, we will tell you about what we have done in each area and, for example, about how this was implemented in the technical document management project from which we started the post.
Processes
In the notorious project on the EDS, the technical documentation management system was deployed along the same scheme on each of 10 objects. The installation includes 4 servers: a database server, an application server, full-text indexing and content management. In the installation they work within a single node, are placed in the data center at the sites. All objects differ slightly in infrastructure, but this does not interfere with global interaction.
At first, according to the DevOps practices, we automated the infrastructures locally, then we brought the supply to the test loop, and then to the customer’s production. Each process worked step by step. The settings of the environments are fixed in the system of source codes, with which the distribution kit is assembled for automatic updating. In case of changes in the configuration, the engineers simply need to make the appropriate changes to the version control system - and then the automatic update will take place without problems.
Thanks to this approach, the testing process is greatly simplified. Earlier in the project were testers, who only did what they manually updated the stands. Now they just come, they look, that everything has earned and are engaged in more useful things. Each update is tested automatically - from the surface level to the automation of the business scenario. The results are laid out in the form of individual reports in TestRail.
Culture
Continuous experimentation is best explained with a test design example. To test the system, which is not yet - creative work. When writing a test plan, you need to understand how to test correctly, which branches to go through. And find a balance between time and budget to determine the optimal number of checks. It is important to choose exactly the necessary tests, consider how the user will interact with the system, take into account the environment and possible external factors. Without continuous experimentation is indispensable.
Now about the culture of interaction. There used to be two opposing sides - engineers and developers. The developers said: “We don’t care how it runs. You are engineers, you are smart, make it so that it can be operated without failures . ” The engineers responded: “You, the developers, are too careless. Let us be careful, and we will put your releases less often. Because every time you put a code through to us, it’s not clear how we interact . ” This is a problem of cultural interaction, which from the point of view of DevOps is built differently. Here you and engineers, and developers are part of a unified team that is aimed at the ever-changing, but at the same time reliable software.
On the scale of one team, specialists are set up to help each other. As it was before? For example, some thick deployment instructions were prepared, for 50 pages. The engineer read it, did not understand something, swore and asked the developer to comment at three in the morning. The developer commented and also cursed - in the end, no one was happy. Plus, naturally there were some mistakes, because I don’t remember everything in the instructions. And now the engineer, together with the developer, is writing a script for the automated deployment of the application software infrastructure. And they talk to each other in almost the same language.
People
The size of the team is determined by the scale of the update. The team is recruited during the formation of the supply, it includes people from the general project team. Then an update plan is written with those responsible for each stage, as the team executes it reports. All team members are interchangeable. As part of the team, we also have a safety net developer, but he almost never has to connect.
Technology
There are few points in the technology scheme, but under them are a lot of technologies - with their descriptions you can release a whole book. So we will highlight the most interesting.
Infrastructure as Code
Now, probably, this concept will not surprise anyone, but earlier the descriptions of infrastructures left much to be desired. Engineers looked at the instructions with horror , the test environments were unique, they were holili and cherished, and dust particles were blown away from them.
Now no one is afraid to experiment. There are basic images of virtual machines, there are ready-made scenarios for deploying environments. All templates and scripts are stored in the version control system and updated promptly. Previously, when it was necessary to deliver a package to the stand, a configuration gap appeared. Now you just need to add a line in the source code.
In addition to infrastructure scripts and pipelines, the Documentation as a Code approach is also used for documentation. Thanks to this, it is easy to connect new people to the project, acquaint them with the system by the functions described, for example, in a test plan, and also reuse test cases.
Continuous delivery and monitoring
In the last article about DevOps, we explained how we chose tools for implementing continuous delivery and monitoring. Often, you do not need to rewrite anything - it is enough to use previously written scripts, build the integration between the components correctly and make a common management console. And all the processes can be started by a single button or schedule.
There are different concepts in English, Continuous Delivery and Continuous Deployment. Both can be translated as “continuous delivery”, but in fact there is a slight difference between them. In our project for the technical documentation of the distributed energy company, rather, the Delivery is used - when the installation for the prod goes on command. In Deployment, the installation is automatic. Continuous Delivery in this project has become a central part of DevOps .
In general, using the collection of certain parameters, you can clearly understand the usefulness of DevOps practices. And bring it to the leadership, which is very fond of numbers. The total number of launches, the execution time of the stages of the script, the proportion of successful launches - all this directly affects everyone’s favorite time to market, that is, the time from committing to the version control system to releasing a version on a production environment. With the introduction of the necessary tools, engineers receive valuable indicators by mail, and the project manager sees them on a dashboard. So you can immediately appreciate the benefits of new tools. And you can try them on to your infrastructure with the help of a DevOps constructor.
Who is our DevOps constructor useful for?
We will not prevaricate: for a start it became useful to us. As we have said, the customer needs to speak the same language, and with the help of the DevOps designer, we can quickly sketch out the basis for such a conversation. Specialists from the business will be able to assess for themselves what they need, and thus develop more quickly. We tried to make the designer as detailed as possible, added a bunch of descriptions so that any user would understand what he chooses.
The format of the designer allows you to take into account the existing developments in the company on building processes, automation. No need to break everything and build again, if you can only choose solutions that are normally integrated into existing processes that can simply fill in the blanks.
Perhaps your development has already moved to a higher level and our tool will seem too "captain". But we find it useful for ourselves and we hope that it will be useful to any of the readers. We remind the link to the designer - if anything, you get the scheme immediately after entering the source data. We will be grateful for the feedback and additions.
Source: https://habr.com/ru/post/455618/
All Articles