Continuous Delivery hecho en Alawar
About a year ago, our team was tasked to start the development of server-side parts of a number of MMO game projects. The specifics of such projects, in addition to the requirements for flexibility, stability and scalability, also include:
Moreover, in the future, our team was supposed to expand, possibly due to outsourcing developers, including for support tasks. Under these conditions, for the successful implementation, it was decided on a par with project versioning, packaging and standardization of a number of development steps to introduce and practice continuous delivery .
The purpose of this article is to tell about the steps taken, the decisions taken and describe the result obtained.
')
Historically, the main language of development of server-side web applications in our company is PHP, so this largely determined the choice of tools.
Summary list:
When choosing a branch model, the “A successful Git branching model” was used, described here with one small difference: it was decided to conduct A / B testing by preparing separate release branches formed from a different set of feature branches. As a result, the role of the develop branch was fully vested in the release branches, and the branch itself disappeared. Otherwise, when creating the next release, we would be forced to include in it all previously released features, which was not always acceptable.
This situation can be demonstrated by the following example. Recall that according to the original:
And let's say that two releases have already been released - release 1.0 with features A and release 2.0 with features A and B, and you need to release release 1.1 with features A and C. Since you develop the branch at the moment it already contains features A and B, then the simplest solution would be to create a feature for branch C from release 1, and then merge it back:
All projects are designed as composer packages.
To reuse the functionality from one project to another, it is widely used to separate some of its separate parts into a separate package.
This is followed by replacing one package with another, dividing one package into two, or transferring functionality from one package to another. Under such conditions, semantic versioning of packages was used for finer control.
This type of versioning is supported in composer using the “~” symbol, for example:
In the case of PHP, we cannot talk about building in the classical sense, as the process of converting project sources into executable code. However, since the main task is still to get ready-to-use software, the name “assembly” is quite correct.
Assembly steps:
To implement the build in the root of each project is an assembly phing script with target'ami:
The assembly script for most projects is the same and differs only in the name of the project: the name attribute, the project tag.
The implementation of automatic testing of projects is done using two frameworks: Behat and PHPUnit.
Using the first provides a significant advantage not only for testing, but also for creating the so-called living documentation . Tests on Gherkin are one of the starting points when getting acquainted with the project of a new programmer, when conducting code review, as well as a number of other works.
Despite familiarity with the materials here and here , we do not have uniform recommendations regarding the depth and detail of these tests, so their content may vary from for example:
to such:
PHPUnit is used not only to implement unit-tests, the presence and contents of which are completely left to the programmer, but also to run Behat tests, using a small workaround '. This makes it possible to run all the tests with one team, as well as have uniform reports on the results of the tests and their coverage of the code.
The build is done using a Jenkins CI server. At the same time, for each releases / XY release branch, a separate assembly task was initiated, which is on the staging environment:
The assembly task itself, as well as the assembly script, were built on the basis of those described here . This is the reason for such a rich list of additional utilities.
In addition to the list of plug-ins listed above, the following has been installed:
It was necessary to find a solution that allows you to simultaneously manage the deployment of several applications, each of which can be installed on several groups of servers (testing, production).
Initially, deployment was carried out by a phing script, which, according to the settings file, performed a number of actions:
It was not very convenient due to the complete lack of support for the installation on remote servers.
After several experiments with Capistrano , Magallanes and other tools, in addition to this script, the console application Installer was implemented. It copies an installation script with the necessary settings to the desired remote server group and executes it there.
Also, this application included the commands for obtaining possible versions of the application and the request for the version installed on the servers (the picture shows the possibility of updating the project in the production environment from version 1.0.19 to 1.0.20):
And the settings file format has been replaced by a more convenient .yml:
This console application was deployed on the build server, and a web interface was made to it in the form of a parameterizable Jenkins build task that executes the console command:
Where,
And this task, in turn, was noted as a downstream project for assembly tasks of release branches using the Parameterized Trigger Plugin .
As a result, we successfully solved the task of implementing the delivery with the following sequence of steps:
The used branching model contributes to the fact that different branches begin to differ greatly from each other, this leads to problems introducing new features into old releases. So far this has not become critical, but there is an idea to try to return the develop branch to develop, and to use other techniques to prepare A / B versions, perhaps something like feature toggles .
There is an interest to try various types of integration with the Jira tracker. Somehow automate for example:
The current running time of the composer is of the order of a few minutes, and a large number of proprietary packages result in a greatly expanded repositories section of the composer.json files. I want to experiment with Satis , to solve these problems.
We have successfully solved the problems we faced:
It should be noted that this scheme is almost not used for the actual updating of projects in production, since it does not cover all the problems of releasing a new one and possibly not compatible with the previous version of the release. Its main application is fast and automated delivery of new functionality to all servers where the application is deployed, and updating in places where it is possible, or not critical - test and intended for closed beta testing of the server.
Suerte!
- the need for A / B testing of different versions of the same game
- opportunity to reuse functionality from one game to another
- high probability of geographic distance from developers involved in the client side of the game
Moreover, in the future, our team was supposed to expand, possibly due to outsourcing developers, including for support tasks. Under these conditions, for the successful implementation, it was decided on a par with project versioning, packaging and standardization of a number of development steps to introduce and practice continuous delivery .
The purpose of this article is to tell about the steps taken, the decisions taken and describe the result obtained.
')
Infrastructure
Historically, the main language of development of server-side web applications in our company is PHP, so this largely determined the choice of tools.
Summary list:
- Git - version control system
- Gitolite - repository management
- Composer - dependency manager
- Phing - assembly and installation scripts
- Jenkins - continuous integration server
- phpunit , behat tests
- phploc , phpmd , pdepend , phpcs , phpcpd , phpcb , phpdox - other utilities
Branching model
When choosing a branch model, the “A successful Git branching model” was used, described here with one small difference: it was decided to conduct A / B testing by preparing separate release branches formed from a different set of feature branches. As a result, the role of the develop branch was fully vested in the release branches, and the branch itself disappeared. Otherwise, when creating the next release, we would be forced to include in it all previously released features, which was not always acceptable.
This situation can be demonstrated by the following example. Recall that according to the original:
This has been the case. It has been branched off.
And let's say that two releases have already been released - release 1.0 with features A and release 2.0 with features A and B, and you need to release release 1.1 with features A and C. Since you develop the branch at the moment it already contains features A and B, then the simplest solution would be to create a feature for branch C from release 1, and then merge it back:
Bundling and Versioning
All projects are designed as composer packages.
To reuse the functionality from one project to another, it is widely used to separate some of its separate parts into a separate package.
This is followed by replacing one package with another, dividing one package into two, or transferring functionality from one package to another. Under such conditions, semantic versioning of packages was used for finer control.
This type of versioning is supported in composer using the “~” symbol, for example:
"require": { ... "alawar/packet-post-process-server": "~1.3", ... }, “Build” project
In the case of PHP, we cannot talk about building in the classical sense, as the process of converting project sources into executable code. However, since the main task is still to get ready-to-use software, the name “assembly” is quite correct.
Assembly steps:
- pumping dependencies through composer
- database migration, - updating only database structure and static data
To implement the build in the root of each project is an assembly phing script with target'ami:
- build - to perform the build steps
- runtests and runtest-with-coverage - for both building and running tests and collecting metrics
The assembly script for most projects is the same and differs only in the name of the project: the name attribute, the project tag.
Testing
The implementation of automatic testing of projects is done using two frameworks: Behat and PHPUnit.
Using the first provides a significant advantage not only for testing, but also for creating the so-called living documentation . Tests on Gherkin are one of the starting points when getting acquainted with the project of a new programmer, when conducting code review, as well as a number of other works.
Despite familiarity with the materials here and here , we do not have uniform recommendations regarding the depth and detail of these tests, so their content may vary from for example:
: # - : | action | uid | | get-user-bonus-code | player1 | "GetUserBonusCodeResponse.txt" # : | action | code | | get-rewards-info | | "template8.txt" to such:
: - PHPUnit is used not only to implement unit-tests, the presence and contents of which are completely left to the programmer, but also to run Behat tests, using a small workaround '. This makes it possible to run all the tests with one team, as well as have uniform reports on the results of the tests and their coverage of the code.
Assembly server
The build is done using a Jenkins CI server. At the same time, for each releases / XY release branch, a separate assembly task was initiated, which is on the staging environment:
- executes the build phing script with the “build runtests-with-coverage” target
- collects test reports and performance utilities
- in the case of an error-free process completion, a new tag of the form $ VERSION_NO. $ BUILD_NUMBER is created in the repository, where $ VERSION_NO is the version number obtained from the branch name, for example, 2.1, and $ BUILD_NUMBER is the assembly sequence number for this assembly task
The assembly task itself, as well as the assembly script, were built on the basis of those described here . This is the reason for such a rich list of additional utilities.
In addition to the list of plug-ins listed above, the following has been installed:
- EnvInject - to initialize the $ VERSION_NO variable
- ChuckNorris and CI-Game - just for fun
Deployment
It was necessary to find a solution that allows you to simultaneously manage the deployment of several applications, each of which can be installed on several groups of servers (testing, production).
Initially, deployment was carried out by a phing script, which, according to the settings file, performed a number of actions:
- created files, folders and symlinks
- did checkout of source codes of the necessary branch / tag
- built the project
- and since each time the checkout was executed in a new folder of the form 2012-01-01T23: 59: 59, I updated the latest symlink, indicating the last deployed version
It was not very convenient due to the complete lack of support for the installation on remote servers.
After several experiments with Capistrano , Magallanes and other tools, in addition to this script, the console application Installer was implemented. It copies an installation script with the necessary settings to the desired remote server group and executes it there.
Also, this application included the commands for obtaining possible versions of the application and the request for the version installed on the servers (the picture shows the possibility of updating the project in the production environment from version 1.0.19 to 1.0.20):
And the settings file format has been replaced by a more convenient .yml:
This console application was deployed on the build server, and a web interface was made to it in the form of a parameterizable Jenkins build task that executes the console command:
/home/projects/installer/installer.phar $command $recipe $environment Where,
- $ command - the name of the command being executed, for example install, status, versions
- $ recipe - code assigned to the version of the project intended for installation
- $ environment - optional name of the server group on which the project should be installed
And this task, in turn, was noted as a downstream project for assembly tasks of release branches using the Parameterized Trigger Plugin .
Eventually
As a result, we successfully solved the task of implementing the delivery with the following sequence of steps:
- the developer makes changes to the release branch
- post-receive hook gitolite initiates the Jenkins assembly task corresponding to this branch
- the build task tests and marks the successful version with a tag
- Jenkins runs the downstream project Installer with the necessary parameters for the project and the group of servers on which the project needs to be updated.
- Installer, successively passing through all the servers of the group, deploys the latest version and updates the latest symlink.
Further
The used branching model contributes to the fact that different branches begin to differ greatly from each other, this leads to problems introducing new features into old releases. So far this has not become critical, but there is an idea to try to return the develop branch to develop, and to use other techniques to prepare A / B versions, perhaps something like feature toggles .
There is an interest to try various types of integration with the Jira tracker. Somehow automate for example:
- creating branches for new tickets of a certain type
- update of status and / or comments to tickets according to test results
- forming change logs
The current running time of the composer is of the order of a few minutes, and a large number of proprietary packages result in a greatly expanded repositories section of the composer.json files. I want to experiment with Satis , to solve these problems.
Conclusion
We have successfully solved the problems we faced:
- to create A / B versions, separate branches are used in the version control system
- package manager allows you to reuse functionality from project to project
- tests on Gherkin and their implementation help to significantly simplify the connection to the project of new developers
- and the continuous delivery scheme described above allows minimizing the time for receiving feedback from the developers of the client-side of the game
It should be noted that this scheme is almost not used for the actual updating of projects in production, since it does not cover all the problems of releasing a new one and possibly not compatible with the previous version of the release. Its main application is fast and automated delivery of new functionality to all servers where the application is deployed, and updating in places where it is possible, or not critical - test and intended for closed beta testing of the server.
Suerte!
Source: https://habr.com/ru/post/202214/
All Articles