Centralized continuous deployment for the year
In one of the previous posts about DevOps, we promised to tell you about the technological component of our CI / CD pipeline.
To describe the whole picture in colors and fully share our emotions (to paint pain and bloody tears), I’ll tell you about what we started a couple of years ago and what we came up with today.
I have been working at Raiffeisen for over five years. At first I was a freelance specialist, and now I lead a division that supports the CI / CD pipeline and develops automation. During this rather big period of time, I and the team managed to work, probably, with 90% of all software used in the bank, to get a lot of useful experience, of which I would like to share a part.
')
As in any large company that did not have an approved set of Middle and Software, we eventually came to a huge stack of technologies used.
Development teams used completely different programming languages, frameworks and tools. Support teams also used what they could do, with varying degrees of success. That is, Dev had his own tools, Ops had his own. If there was automation, then at the level “cmd calls the batch file, it launches a VBS script that creates an object in COM +, which ...” NO, EVERYTHING, I DO NOT WANT TO RECOMMEND!
So, about technology. Let's start with the CD, if you can call it that.
Several subversion installations, in common - SVN, a couple of underground installations of GIT, it all revolves either on a wonderful Win 2003, or on a new <ironic> RHEL 5. And of course it is not related to each other, from the word NO. Many preferred not to use version control systems at all. The knowledge and documentation base was conducted either in the same SVN (yes) in the form of msdoc instructions, or on some internal wiki.
It should be noted that there were 3-4 teams that were actively building their automated process. What could not but rejoice. But their experience was difficult to scale because of the number and differences in the technologies used. Jenkins, TeamCity, Bamboo, Artifactory, Nexus, in general, everyone did what he wanted and how he could.
All these tools were supported by the developers themselves, who spent some of their time on it instead of cutting new features.
Remedy was used as a ticket management system, and this is my personal, terrible, fierce pain. It seems to me that those who worked will understand, and if you are lucky enough not to deal with Remedy, I can only envy.
The test management system was no less beautiful - HP ALM. As a bug tracking system, I have no complaints about it, the whole trouble is in integration. But it is possible that this is just my personal “love” for HP software products.
The first major change in our zoo, from my point of view, was the appearance of Jira. After Remedy it was a breath of fresh air: easy, fast, comfortable! At first, Jira decided to try several development teams, and as a result, it became a general banking system. At first, all work on non-industrial environments was transferred to it, and now we are introducing JIRA SD as a system for managing incidents and changes. It is also important that JIRA began to work together on the tasks of both Dev and Ops. Not long to wait and Confluence.
Then we took on the centralization of all source code storage systems. We lifted the centralized SVN booth, into which the repositories of all teams were moved. Also, as an alternative, SVN raised Bitbucket, and many teams immediately went there. As a result, the entire self-written code was stored in two products, and each team decided on their own what to use.
By this time, the idea of ​​creating a full-fledged pipeline for CI / CD automation has already matured. The hottest disputes were around choosing a tool to automate the assembly and installation of software, as well as a tool for storing binary objects. We have been looking at what was on the market for a very long time, compared, read, studied, tried and tested something, including a self-written tool developed at our head office in Vienna.
As a result, for several reasons, we stopped at Bamboo:
Talk a little about the architecture of Bamboo. It uses an application server with software, a server with a database, and so-called Bamboo agents. That is, it is a set of servers (the amount depends on the purchased license) on which the software is installed, which is necessary for your assemblies and installations. When creating an assembly plan, you need to specify the necessary components in the requirements. When starting the assembly, a free agent is searched that meets your requirements, and the application is built on it. It develops on the same principle. In this case, the assembly and deployment are performed equally in all environments, industrial and non-industrial!
Artifactory was chosen as a tool for storing binary objects. It took us about a month to compare and choose between the Nexus and the Artifactory. Different sources on the Internet talked about the superiority of one or the other tool. We were both suitable, but in the end the licensing policy of Jfrog won out. And we still have never regretted our choice, the tool is convenient, actively developing.
For a couple of months, we have raised and integrated non-industrial and industrial pipelines. RHEL 7 was chosen as the OS for all products without much thought, and PostgreSQL was chosen as the DBMS.
The exciting process of moving teams to a centralized CI / CD pipeline has begun. The more they passed, the more software needed to be installed on the Bamboo agents. Let me explain: the installation and debugging of about 45 components on 10 agents took about 5 months.
This whole story prompted us to revise the majority of intra-team and part of the general banking processes. But optimization alone did not help to achieve the desired speed of work. And once we realized that it became impossible to maintain and install the necessary software commands in manual mode. We discussed with the developers, suspended the installation of new and updated current software, and went to learn automation.
Considering that with time Bamboo agents will become much more than 25, they decided to use a bunch of Bamboo and Ansible to create and update agents. Than Ansible for us was more interesting than other configuration management tools:
It took about two months to prepare the first beta scripts for RHEL agents. With Windows, as always, there were a lot of problems, we have already spent 3-4 months here. At the same time, we are still at war with nuget and chocolatey. But then the creation and configuration of the agent now takes only two hours instead of two weeks. By the way, today we have ~ 50 agents, and the number of installed components is approaching 120. It would seem that here it is happiness, two hours - and the new agent is ready! But no. Creating a new server is no less a wonderful story, considering the number of teams involved in this process. With all the approvals and change management, we spent about a week creating a new RHEL server. And then another couple of days to transfer the root.
We had to change something.
Began to think. In the meantime, SonarQube was added to the pipeline - a tool for static code analysis and technical debt management. Cool software!
But back to our sheep: in the era of IaaS it was very painful to realize that we are creating a virtual machine for a week. Why not use AWS / GoogleCloud / Azure or any other cloud? The answer is simple - the legislation of the Russian Federation forbids, and our security service was not thrilled (to put it mildly) from the idea of ​​using an external cloud.
Then make your own!
The bank had already experimented with vRealize by that time, we jumped at the opportunity and joined the experiment. As a result, we received a pool of resources that could be managed and created from templates servers with the necessary OS.
To create a dozen servers instead of a week, an hour began to go away, and our joy knew no bounds.
But we decided not to stop at this. If you automate, then you need to learn how to create servers in vRealize of Bamboo. A week of research and another week to implement your own plugin - and the first server was created directly from the Bamboo deploy project. By that time, the people decided that we didn’t need SVN at all, and with it Fisheye + Crucible: Bitbucket has many more features, it is more modern and more convenient. We helped the teams move to Bitbucket and turned off the "out-of-date" SVN.
What else to automate?
Dev / Ops / DevOps / BizDevOps - what prevents all these people from having automated assembly and installation? Work with change management system requests! But how can we be without change management, it's an important thing! <slyly] If you look at the situation more closely, it becomes clear that what is preventing is not the process itself, but the need to plan, coordinate and record the result of the change MANUALLY!
And once again comes the opportunity to write your own plug-ins for Bamboo! A week of research, two weeks for implementation, and we are creating from the same deploy project the first pre-agreed CRQ in an industrial environment.
We launched a pilot use of several tools for ChatOps. Dragging over HipChat, because despite all its flaws, I haven’t yet found a better onpremise solution. Again, by using a large stack of Atlassian products, we’ll get a bunch of buns. For example, the integration of all tools in the Jira-task allows you to track all commits, assemblies and deployments performed during the implementation of this feature. Incorporating HipChat into the process will allow, for example, setting a specific release to a specific environment, running autotests, returning the result to chat and sending notifications to users that you can start UAT by entering one command in the chat window ...
Our pipeline now looks like this:
The mutual integration of all these applications allows us to automate almost all routine activities related to software production. We have more time to improve the quality of our products or for some research.
We promise to return and share a new interesting experience.
To describe the whole picture in colors and fully share our emotions (to paint pain and bloody tears), I’ll tell you about what we started a couple of years ago and what we came up with today.
About myself
I have been working at Raiffeisen for over five years. At first I was a freelance specialist, and now I lead a division that supports the CI / CD pipeline and develops automation. During this rather big period of time, I and the team managed to work, probably, with 90% of all software used in the bank, to get a lot of useful experience, of which I would like to share a part.
')
What was the part of the technology used and the approach two years ago and what we came to
As in any large company that did not have an approved set of Middle and Software, we eventually came to a huge stack of technologies used.
Development teams used completely different programming languages, frameworks and tools. Support teams also used what they could do, with varying degrees of success. That is, Dev had his own tools, Ops had his own. If there was automation, then at the level “cmd calls the batch file, it launches a VBS script that creates an object in COM +, which ...” NO, EVERYTHING, I DO NOT WANT TO RECOMMEND!
So, about technology. Let's start with the CD, if you can call it that.
Several subversion installations, in common - SVN, a couple of underground installations of GIT, it all revolves either on a wonderful Win 2003, or on a new <ironic> RHEL 5. And of course it is not related to each other, from the word NO. Many preferred not to use version control systems at all. The knowledge and documentation base was conducted either in the same SVN (yes) in the form of msdoc instructions, or on some internal wiki.
It should be noted that there were 3-4 teams that were actively building their automated process. What could not but rejoice. But their experience was difficult to scale because of the number and differences in the technologies used. Jenkins, TeamCity, Bamboo, Artifactory, Nexus, in general, everyone did what he wanted and how he could.
“We will build our pipeline, with blackjack and courtesans”
All these tools were supported by the developers themselves, who spent some of their time on it instead of cutting new features.
Remedy was used as a ticket management system, and this is my personal, terrible, fierce pain. It seems to me that those who worked will understand, and if you are lucky enough not to deal with Remedy, I can only envy.
The test management system was no less beautiful - HP ALM. As a bug tracking system, I have no complaints about it, the whole trouble is in integration. But it is possible that this is just my personal “love” for HP software products.
The first major change in our zoo, from my point of view, was the appearance of Jira. After Remedy it was a breath of fresh air: easy, fast, comfortable! At first, Jira decided to try several development teams, and as a result, it became a general banking system. At first, all work on non-industrial environments was transferred to it, and now we are introducing JIRA SD as a system for managing incidents and changes. It is also important that JIRA began to work together on the tasks of both Dev and Ops. Not long to wait and Confluence.
Then we took on the centralization of all source code storage systems. We lifted the centralized SVN booth, into which the repositories of all teams were moved. Also, as an alternative, SVN raised Bitbucket, and many teams immediately went there. As a result, the entire self-written code was stored in two products, and each team decided on their own what to use.
By this time, the idea of ​​creating a full-fledged pipeline for CI / CD automation has already matured. The hottest disputes were around choosing a tool to automate the assembly and installation of software, as well as a tool for storing binary objects. We have been looking at what was on the market for a very long time, compared, read, studied, tried and tested something, including a self-written tool developed at our head office in Vienna.
As a result, for several reasons, we stopped at Bamboo:
- Out of the box other buns in integrations with Atlassian products.
- Missing features are 90% covered by available plugins.
- There is a prepared, developed SDK for writing your own plugins.
- Support from a major vendor.
Talk a little about the architecture of Bamboo. It uses an application server with software, a server with a database, and so-called Bamboo agents. That is, it is a set of servers (the amount depends on the purchased license) on which the software is installed, which is necessary for your assemblies and installations. When creating an assembly plan, you need to specify the necessary components in the requirements. When starting the assembly, a free agent is searched that meets your requirements, and the application is built on it. It develops on the same principle. In this case, the assembly and deployment are performed equally in all environments, industrial and non-industrial!
Artifactory was chosen as a tool for storing binary objects. It took us about a month to compare and choose between the Nexus and the Artifactory. Different sources on the Internet talked about the superiority of one or the other tool. We were both suitable, but in the end the licensing policy of Jfrog won out. And we still have never regretted our choice, the tool is convenient, actively developing.
For a couple of months, we have raised and integrated non-industrial and industrial pipelines. RHEL 7 was chosen as the OS for all products without much thought, and PostgreSQL was chosen as the DBMS.
The exciting process of moving teams to a centralized CI / CD pipeline has begun. The more they passed, the more software needed to be installed on the Bamboo agents. Let me explain: the installation and debugging of about 45 components on 10 agents took about 5 months.
This whole story prompted us to revise the majority of intra-team and part of the general banking processes. But optimization alone did not help to achieve the desired speed of work. And once we realized that it became impossible to maintain and install the necessary software commands in manual mode. We discussed with the developers, suspended the installation of new and updated current software, and went to learn automation.
Automation support team can't automate! WTF ?!
Considering that with time Bamboo agents will become much more than 25, they decided to use a bunch of Bamboo and Ansible to create and update agents. Than Ansible for us was more interesting than other configuration management tools:
- ease of learning,
- lack of server side
- Well, all the other advantages of this software.
It took about two months to prepare the first beta scripts for RHEL agents. With Windows, as always, there were a lot of problems, we have already spent 3-4 months here. At the same time, we are still at war with nuget and chocolatey. But then the creation and configuration of the agent now takes only two hours instead of two weeks. By the way, today we have ~ 50 agents, and the number of installed components is approaching 120. It would seem that here it is happiness, two hours - and the new agent is ready! But no. Creating a new server is no less a wonderful story, considering the number of teams involved in this process. With all the approvals and change management, we spent about a week creating a new RHEL server. And then another couple of days to transfer the root.
With great power comes great responsibility.
We had to change something.
Began to think. In the meantime, SonarQube was added to the pipeline - a tool for static code analysis and technical debt management. Cool software!
But back to our sheep: in the era of IaaS it was very painful to realize that we are creating a virtual machine for a week. Why not use AWS / GoogleCloud / Azure or any other cloud? The answer is simple - the legislation of the Russian Federation forbids, and our security service was not thrilled (to put it mildly) from the idea of ​​using an external cloud.
Then make your own!
The bank had already experimented with vRealize by that time, we jumped at the opportunity and joined the experiment. As a result, we received a pool of resources that could be managed and created from templates servers with the necessary OS.
To create a dozen servers instead of a week, an hour began to go away, and our joy knew no bounds.
But we decided not to stop at this. If you automate, then you need to learn how to create servers in vRealize of Bamboo. A week of research and another week to implement your own plugin - and the first server was created directly from the Bamboo deploy project. By that time, the people decided that we didn’t need SVN at all, and with it Fisheye + Crucible: Bitbucket has many more features, it is more modern and more convenient. We helped the teams move to Bitbucket and turned off the "out-of-date" SVN.
What else to automate?
Dev / Ops / DevOps / BizDevOps - what prevents all these people from having automated assembly and installation? Work with change management system requests! But how can we be without change management, it's an important thing! <slyly] If you look at the situation more closely, it becomes clear that what is preventing is not the process itself, but the need to plan, coordinate and record the result of the change MANUALLY!
And once again comes the opportunity to write your own plug-ins for Bamboo! A week of research, two weeks for implementation, and we are creating from the same deploy project the first pre-agreed CRQ in an industrial environment.
What do we do now?
We launched a pilot use of several tools for ChatOps. Dragging over HipChat, because despite all its flaws, I haven’t yet found a better onpremise solution. Again, by using a large stack of Atlassian products, we’ll get a bunch of buns. For example, the integration of all tools in the Jira-task allows you to track all commits, assemblies and deployments performed during the implementation of this feature. Incorporating HipChat into the process will allow, for example, setting a specific release to a specific environment, running autotests, returning the result to chat and sending notifications to users that you can start UAT by entering one command in the chat window ...
Our pipeline now looks like this:
The mutual integration of all these applications allows us to automate almost all routine activities related to software production. We have more time to improve the quality of our products or for some research.
We promise to return and share a new interesting experience.
Source: https://habr.com/ru/post/342126/
All Articles