About structuring and automation
I would like to share some experience in the automation and standardization of the development process. The article does not claim to be true, but I hope it will be an interesting piece of reading for the end of the working week.
To begin with, I’ll tell you a little about how it all began:
The biggest problem was the lack of any mechanisms to control consistency in functionality between the components of the system. There was not a single criterion, based on which, one could safely say that the two components of the system taken at some point in time are consistent with each other.
Coordinated by functionality, we call the components that communicate with each other in the same language of the same protocol version. In our case, the protocol version is also closely intertwined with business logic. More precisely, no new business logic can be introduced without changing or extending the protocol.
')
During the long debate on “how to conduct svn correctly”, the option of tagging was chosen and a number of requirements / rules were formulated which should be followed:
Trunk
Tag N
Disclaimer: in the case of a new component of the system, which can already be started to test, but which clearly does not meet the functionality of the last tag, in order to get into the overall testing process, such a component can be taken from the trunk to the responsibility of the developer and the general understanding that absolutely exceptional and temporary situation, until a new tag appears.
Our tags are not so tags. But they are closer in essence precisely to the concept of “tag” rather than to the concept of “brunch”.
There was also another SVN reference format under discussion - to conduct the main development in the branches. if we have a major change, we create a separate brunch, implement this change there and merge it into the trunk. In this case, the trunk should always be stable, so that at any time you can take all the components of the system and they were at least coordinated in functionality, as ready as possible for production. In our case, this does not look viable. Since sometimes one major change can come out at the time of the implementation of another. Then I would have to create a separate brunch for him and switch to it. Problems would begin at the moment when it would be necessary to smuggle two branches into a trunk.
We have a very long development cycle, and the process of transition from version to version is not yet tested. Therefore, by giving our product to public testing, we would like to be able to do hot fixes. But without a tag (branch) for each final version, it would not be possible to do this. Also, we are not able to say that “they say it will be fixed in the next version”.
Further in the article I will use the term “tag” precisely in the meaning that I described above.
From the tags there would be no benefit if all components were coordinated in functionality. The bottleneck in this case is the point in time when wecut the scalp to create a tag. As a rule, at one of the meetings we decide that within the next week we should prepare a trunk for creating the tag. That is, each of the projects must come to a state when it is coordinated in functionality with all the components with which it communicates directly. After that we create a tag. Which goes into testing. Tag we call in a special format MAJOR_NUM.MINOR_NUM.
The next important step was the introduction of a criterion based on which one could safely say that the two components are consistent with each other. We decided that this criterion would be the version. In this case, the version itself must carry information based on which it can be understood that the two components are consistent.
Version format: MAJOR_NUM.MINOR_NUM.REVISION. Those. The first part of the version is also the tag number. And we know that all the components in the tag are consistent. Specifying two numbers at once in the tag name allows you to make a normal transition between major versions, i.e. instead of 1.1.x, 1.2.x, 1.3.x, 2.4.x , we will have 1.1.x, 1.2.x, 1.3.x, 2.1.x. The revision number remains and is used normally. This is the revision number of each of the individual components, not just the last revision of the tag.
The next step was the process of implementing CI (Continuous Integration). The choice fell on, I think many well-known CI system, Jenkins. The choice was between Cruise control, with which I have not the most pleasant experience, Team City which is paid for its normal use and directly Jenkins, which is free, widely distributed and well documented. The free factor was important because it was scary to ask the authorities for money for something that is not a fact that they would take root.
The components of our system are very diverse and currently use 3 platforms for building: windows, linux and mac os. It turns out one master and 2 slave (I do not want to repeat the word "slave" anymore). The question of where to do the wizard did not arise, of course, on Linux. The initial configuration from one master (Linux) and one slave (Windows) was deployed. The process of automating the assembly of all our components has begun. In the course of work, the tag 1.1 has already appeared and the number of tasks in Jenkins has exceeded 10. And then the problems started. One day, Jenkins fell. We raised him, he fell again. Once again. And it continued to fall, no matter how we run it through Tomcat or as a service. Regardless of JAVA machine, open JDK or SUN java. Regardless of the current version. Rather, it was not Jenkins himself who fell, but the JAVA car that he was demolishing. I even started a separate bug in their bug track ( JENKINS-16199 , but that’s all stalled). Having spent a week and a half on any attempts to understand what is happening, or at least why, it was decided to transfer the master to the Windows system. And about a miracle, in the same configuration everything began to work stably. Since the time the number of tasks has doubled, but everything continues to work. As a result, the final structure looks like this:
Master on Windows, two Slave, on Linux and MacOS, interaction over SSH through public keys.
We have expanded the standard Jenkins delivery with the following plugins:
Copy Artifact Plugin - We use to use an assembly artifact in another. There is one simple point where you can stumble.
If you save an artifact in one of the tasks like this
And in another, using this plugin, download this art.
Then at least you should indicate the path to it as well as in the first case, but it still does not hurt to set the “Flatten directories” option to discard unnecessary paths and copy only the art itself to the place where you need it.
Jenkins description setter plugin - Allows you to specify some information about each build in the Build Description field. Those tasks that have a version, bring it there.
Extra Columns Plugin - allows you to add a Build Description field to the View.
Publish Over SSH - A great plugin for uploading files and executing commands on a remote machine. True, we are not using it yet due to the fact that autotesting has not been established yet. The disadvantage of this plugin is that it requires a preliminary configuration of all SSH connections and each task clearly indicates which connection to use at the time of its creation.
Python Plugin - part of the build steps written in Python, because it's easier. Including there is a separate task that does almost everything the same as Publish Over SSH but only allowing you to configure the access parameters to a remote machine on the fly.
Xcode integration - Used to build a project on MacOS. We had to sweat in order to be able to do builds for different versions of iOS. As a result, the build machine was allowed to use the sudo xcode-select -switch command without entering a password.
I have already mentioned the need to remotely fill and execute commands, which the Publish Over SSH plugin almost perfectly solves. But which does not allow directly for each build process to specify the desired target.
We have our own installer, several testers and several machines. Many people are too lazy to drag this file to themselves and do something with it to install the product. For this purpose the “Remote deploy” coupling task was created.
And so that she can:
Pick up the installer from the tag that the user will specify
Install it on the machine that the user specifies.
Here's what it looks like.
This task is parameterized.
Options
TAG, HOST, USER, DEPLOY_DIR, OPENSSL_DIR, there is no magic, normal lines.
JOBS_NUMBER - “Build selector for Copy Atrifact”, allows you to specify which build version to take, the last successful one or some specific number or something else.
PASSWORD is a parameter of the “Password Parameter” type - closes the password with asterisks.
This is how the necessary assembly is taken from the necessary task.
Views in Jenkins are a means of customizing the display of projects. We removed one column and added a “Build Description” for our species Trunk, Tag_X. We would like the Jenkins default view to be the same. But Jenkins does not allow such manipulations with his main appearance. Instead, you can create your own view (Default), add all the projects by mask. * And make this view the default view.
We have a special script for creating a version and all its possible variations. In the beginning, he appeared in each of the projects in which he needed similar functionality. After a while, we put it in one place and wherever it is needed, we pull it as an external dependence. And then Epic fail happened when he was accidentally corrected in one of the projects where he was an addiction. In Jenkins lined up 15 projects for reassembly, idle reassembly.
The first thing that came to mind was to block the file for changes. The needs-lock property was hung on the file and it was blocked by the user’s special build (svn lock).
But there is a better solution. Much better. Jenkins allows you to enter filtering before running the assembly initiated by changing the repository. This is in the section “Source Control -> Advanced”. We are interested in the “Excluded Regions” parameter in which you can indicate changes in which files and directories should be ignored.
This is where my story ends. At the moment I have already left the place of work where this experience was obtained and this article was written, so I can hardly answer the possible questions about the details and settings.
Thanks for attention.
FrimInc , zzapp for reading the article
In the beginning was the word
To begin with, I’ll tell you a little about how it all began:
- The repository in which there was only one main branch.
- 5 projects that need to interact in a single ecosystem. And a few more on the way.
- Constantly supplemented requirements, leading to irreversible changes in the protocol of communication between all components (not the best situation, but it was already).
- Each project had its own version format. Some were based on revisions, others on a date, and the third on that and that.
- The lack of any consistency between all components (was only in words).
- The absence of any automation processes (something still was, but not a lot).
- The lack of a single repository of all distributions (some were stored in svn, others in Track Studio, the third was reassembled from source code each time (it was fun to watch how testers unloaded application sources from svn and made make clean all to get a new binary for testing)).
In the beginning was structuring
The biggest problem was the lack of any mechanisms to control consistency in functionality between the components of the system. There was not a single criterion, based on which, one could safely say that the two components of the system taken at some point in time are consistent with each other.
Coordinated by functionality, we call the components that communicate with each other in the same language of the same protocol version. In our case, the protocol version is also closely intertwined with business logic. More precisely, no new business logic can be introduced without changing or extending the protocol.
')
During the long debate on “how to conduct svn correctly”, the option of tagging was chosen and a number of requirements / rules were formulated which should be followed:
Trunk
- may not be agreed upon by functionality
- not subject to manual comprehensive testing
- not subject to rollout to production or merge to branch
Tag N
- all components are consistent in functionality
- functionality does not change
- involved in the main testing process
- May be marked as Production after testing.
- Can be frozen and forgotten (deleted) after functional testing.
- Bug Fix in Trunk
- From Trunk only critical bugs revealed in earlier versions
Disclaimer: in the case of a new component of the system, which can already be started to test, but which clearly does not meet the functionality of the last tag, in order to get into the overall testing process, such a component can be taken from the trunk to the responsibility of the developer and the general understanding that absolutely exceptional and temporary situation, until a new tag appears.
Our tags are not so tags. But they are closer in essence precisely to the concept of “tag” rather than to the concept of “brunch”.
There was also another SVN reference format under discussion - to conduct the main development in the branches. if we have a major change, we create a separate brunch, implement this change there and merge it into the trunk. In this case, the trunk should always be stable, so that at any time you can take all the components of the system and they were at least coordinated in functionality, as ready as possible for production. In our case, this does not look viable. Since sometimes one major change can come out at the time of the implementation of another. Then I would have to create a separate brunch for him and switch to it. Problems would begin at the moment when it would be necessary to smuggle two branches into a trunk.
We have a very long development cycle, and the process of transition from version to version is not yet tested. Therefore, by giving our product to public testing, we would like to be able to do hot fixes. But without a tag (branch) for each final version, it would not be possible to do this. Also, we are not able to say that “they say it will be fixed in the next version”.
Further in the article I will use the term “tag” precisely in the meaning that I described above.
We cut the scalps
From the tags there would be no benefit if all components were coordinated in functionality. The bottleneck in this case is the point in time when we
Single version format
The next important step was the introduction of a criterion based on which one could safely say that the two components are consistent with each other. We decided that this criterion would be the version. In this case, the version itself must carry information based on which it can be understood that the two components are consistent.
Version format: MAJOR_NUM.MINOR_NUM.REVISION. Those. The first part of the version is also the tag number. And we know that all the components in the tag are consistent. Specifying two numbers at once in the tag name allows you to make a normal transition between major versions, i.e. instead of 1.1.x, 1.2.x, 1.3.x, 2.4.x , we will have 1.1.x, 1.2.x, 1.3.x, 2.1.x. The revision number remains and is used normally. This is the revision number of each of the individual components, not just the last revision of the tag.
Then came the automation
The next step was the process of implementing CI (Continuous Integration). The choice fell on, I think many well-known CI system, Jenkins. The choice was between Cruise control, with which I have not the most pleasant experience, Team City which is paid for its normal use and directly Jenkins, which is free, widely distributed and well documented. The free factor was important because it was scary to ask the authorities for money for something that is not a fact that they would take root.
I am your master you are my slave
The components of our system are very diverse and currently use 3 platforms for building: windows, linux and mac os. It turns out one master and 2 slave (I do not want to repeat the word "slave" anymore). The question of where to do the wizard did not arise, of course, on Linux. The initial configuration from one master (Linux) and one slave (Windows) was deployed. The process of automating the assembly of all our components has begun. In the course of work, the tag 1.1 has already appeared and the number of tasks in Jenkins has exceeded 10. And then the problems started. One day, Jenkins fell. We raised him, he fell again. Once again. And it continued to fall, no matter how we run it through Tomcat or as a service. Regardless of JAVA machine, open JDK or SUN java. Regardless of the current version. Rather, it was not Jenkins himself who fell, but the JAVA car that he was demolishing. I even started a separate bug in their bug track ( JENKINS-16199 , but that’s all stalled). Having spent a week and a half on any attempts to understand what is happening, or at least why, it was decided to transfer the master to the Windows system. And about a miracle, in the same configuration everything began to work stably. Since the time the number of tasks has doubled, but everything continues to work. As a result, the final structure looks like this:
Master on Windows, two Slave, on Linux and MacOS, interaction over SSH through public keys.
About implants
We have expanded the standard Jenkins delivery with the following plugins:
Copy Artifact Plugin - We use to use an assembly artifact in another. There is one simple point where you can stumble.
If you save an artifact in one of the tasks like this
And in another, using this plugin, download this art.
Then at least you should indicate the path to it as well as in the first case, but it still does not hurt to set the “Flatten directories” option to discard unnecessary paths and copy only the art itself to the place where you need it.
Jenkins description setter plugin - Allows you to specify some information about each build in the Build Description field. Those tasks that have a version, bring it there.
Extra Columns Plugin - allows you to add a Build Description field to the View.
Publish Over SSH - A great plugin for uploading files and executing commands on a remote machine. True, we are not using it yet due to the fact that autotesting has not been established yet. The disadvantage of this plugin is that it requires a preliminary configuration of all SSH connections and each task clearly indicates which connection to use at the time of its creation.
Python Plugin - part of the build steps written in Python, because it's easier. Including there is a separate task that does almost everything the same as Publish Over SSH but only allowing you to configure the access parameters to a remote machine on the fly.
Xcode integration - Used to build a project on MacOS. We had to sweat in order to be able to do builds for different versions of iOS. As a result, the build machine was allowed to use the sudo xcode-select -switch command without entering a password.
Special black magic
I have already mentioned the need to remotely fill and execute commands, which the Publish Over SSH plugin almost perfectly solves. But which does not allow directly for each build process to specify the desired target.
We have our own installer, several testers and several machines. Many people are too lazy to drag this file to themselves and do something with it to install the product. For this purpose the “Remote deploy” coupling task was created.
And so that she can:
Pick up the installer from the tag that the user will specify
Install it on the machine that the user specifies.
Here's what it looks like.
This task is parameterized.
Options
TAG, HOST, USER, DEPLOY_DIR, OPENSSL_DIR, there is no magic, normal lines.
JOBS_NUMBER - “Build selector for Copy Atrifact”, allows you to specify which build version to take, the last successful one or some specific number or something else.
PASSWORD is a parameter of the “Password Parameter” type - closes the password with asterisks.
This is how the necessary assembly is taken from the necessary task.
Customize View
Views in Jenkins are a means of customizing the display of projects. We removed one column and added a “Build Description” for our species Trunk, Tag_X. We would like the Jenkins default view to be the same. But Jenkins does not allow such manipulations with his main appearance. Instead, you can create your own view (Default), add all the projects by mask. * And make this view the default view.
We shoot at idle
We have a special script for creating a version and all its possible variations. In the beginning, he appeared in each of the projects in which he needed similar functionality. After a while, we put it in one place and wherever it is needed, we pull it as an external dependence. And then Epic fail happened when he was accidentally corrected in one of the projects where he was an addiction. In Jenkins lined up 15 projects for reassembly, idle reassembly.
The first thing that came to mind was to block the file for changes. The needs-lock property was hung on the file and it was blocked by the user’s special build (svn lock).
But there is a better solution. Much better. Jenkins allows you to enter filtering before running the assembly initiated by changing the repository. This is in the section “Source Control -> Advanced”. We are interested in the “Excluded Regions” parameter in which you can indicate changes in which files and directories should be ignored.
To prison
This is where my story ends. At the moment I have already left the place of work where this experience was obtained and this article was written, so I can hardly answer the possible questions about the details and settings.
Thanks for attention.
Thanks
FrimInc , zzapp for reading the article
Source: https://habr.com/ru/post/197158/
All Articles