Taming "dinosaur", or CI / CD and IBM System i
I want to share with you the story of how we built CI / CD processes for software written for a non-standard platform, which many consider to be a “dinosaur” - IBM System i aka AS / 400.
AS / 400 is used by many large Russian banks, and all of them are slowly moving towards CI / CD. Raiffeisenbank was one of the first (if not the first) to use these practices to install software on the AS / 400 platform in commercial operation.
We went a long way to automating installations; we were pushed for various reasons:
After several unsuccessful installations, we decided to include a special installer in any installation on the AS / 400. For each specific delivery, the developers prepared a CL-script. He checked the completeness of the delivery, the availability of a backup of the objects being changed and the installation of the correct authorization on the objects, and also installed the software himself and handled abnormal situations.
')
The new approach was liked by the support service and the business, because the stability of the installations has improved. But it became more difficult for developers, because instead of a simple instruction like “take object xxx, move to another environment, put in yyy, add standard authorization” , I had to write an additional program for each installation.
Some problems remained unresolved. Tested software did not always correspond to what was put into productive. The developer could always change something in the prepared program at the last moment, guided by his vision of the correctness of the system. This could lead to disasters in the industrial exploitation environment.
In addition, it always took us a long time to move from stage to stage. After writing the code, the developer asked the tester: "Where to put something?"; after a while the tester answered and transferred the task to the developer; he set and sent the task back. Then “throwing the ball” began when correcting the found defects, while preparing the delivery for going into battle. At best, several minutes were spent on each such transition, at worst - days.
The support service was extremely important to understand for which task the productive objects were last changed, how to roll back the system to the previous version, in which non-industrial environments the change was made and what it was.
With the emergence of CI / CD and DevOps movements in our bank, we received new tools and immediately began to master them.
The source code has moved to the appropriate repository — Git (Bitbucket), collected supplies — to Artifactory, and the build, installation, and testing process — to Bamboo.
If the RPG / C / CL source codes more or less smoothly fit into the code storage, because this is just text, then I had to tinker with the orchestrator of assemblies and installations (Bamboo), because standard Bamboo plug-ins for AS / 400 do not exist at the moment.
We began to try various installation options. The concepts were completely opposite, starting with “let's store all the installation and assembly logic in Bamboo as ftp / ssh scripts, without any additional code” , ending “but let's make a universal installer on AS / 400”.
As a result, we have a process that is practically no different from CI / CD processes for any other platform:
The central part of the process is Jira, it gathers all information about requirements, documentation, commits, assemblies and installations.
All the requirements appear in our form as Jira-tasks and get into the work of analysts who find out and describe the requirements and scenarios for using the software, after which the developers pull themselves to the task.
Developers write code and publish it in Bitbucket. The commit identifier and link to it automatically appears in the Jira task.
The commit in Bitbucket automatically launches the application build in Bamboo and the publication of the collected delivery in the Artifactory.
When the build is successfully completed, it launches a software installation plan in the developers' sandbox, where they can check what they have done. Also, after a successful installation, a set of autotests is launched, informing the developer about the quality of the new delivery. Information about the status of installations is immediately reflected in both the Jira task and Bitbucket.
If there are no defects, then the developer can proceed to the next task.
Then the tester is included in the process, who decides for himself where to install the testing software, and puts it by pressing a button.
After testing is completed, a “change request” is created for support professionals who validate and agree on the change. After that, the attendants at the right moment press the button for installation in Bamboo.
By the way, in one of the previous articles we wrote about our application, which is sensitive to long-term inaccessibility and processes millions of requests per day. It goes into the industrial environment only in the process described above.
Now I suggest a little immersed in the details of the assembly and installation on the AS / 400.
Unfortunately, it is impossible to compile RPG / CL programs on * nix or Windows machines, so we had to complicate the standard software build scheme a bit:
For ourselves, we immediately identified the “reference” environment of AS / 400, on which Bamboo agents do a remote compilation.
The developer, creating new software, takes care of how it will be assembled and installed in the future. To do this, each software in Bamboo has its own Build and Deploy scripts.
After the commit, the Bamboo application build plan is automatically launched, configured for a specific repository.
The source code is downloaded and hosted on the build agent, and then FTP is put into a temporary build library on the AS / 400. Then ssh is compiled and the application build program is called. The task of the collector is to compile all the programs and the installer, and then put them in one package, ready for deployment in any environment.
Typical software package ready for installation:
Then the finished FTP package is transferred from AS / 400 to the build agent.
Here it should be noted that changing programs is only one part of the change, and the second is scripts for changing the database. They move from Bitbucket to the build agent, and join as another artifact to the already assembled package of AS / 400 programs.
After the build is complete, artifacts are published in Bamboo and Artifactory.
Bamboo's build plans are designed to be used for any AS / 400 software written in a bank. To add a new plan, we need 15 minutes - this takes into account the cloning of any other plan and changing the values of some variables.
One of the main advantages of our installation process is that it is repeatable and the same for any environment, be it a production environment or a development environment. We start testing the installation in the product already at the installation stage in the development environment.
In general, the installation looks like this:
By triggering or pressing a button, the deployment plan in Bamboo for the specified environment is launched. First of all, our artifacts are downloaded from the Artifactory to the installation agent - these are Liquibase-scripts and the installation package of programs (save-file).
Then, save file objects via FTP are moved and unpacked into a temporary library on the AS / 400, in an environment where the programs will be installed. Next, the Liquibase utility is launched, which, using the jt400 library, connects to the AS / 400 and executes the database update script, recording the corresponding entries in the change log.
Finally, Bamboo via ssh calls the installer, which backs up the objects to be changed, checks the contents of the delivery, sets the change and adjusts the authorization.
The installation result is sent to all team members subscribed to the list:
We added a small useful feature: the updated version of the software at the program level is assigned the task number, within which the last change was made, and the Bitbucket committee ID:
This helps to quickly find the latest change and see the changes in the code. In emergency situations, even the support service can make small changes to the code and send the fix to the product without having valuable knowledge about compiling, building and installing software.
We currently run about 150 installations in AS / 400 environments per day. We are constantly improving the process, it is not knocked out in stone and is not fixed in the standards and procedures - it lives and develops. Developers are coming to taste more and more, they have new ideas, they offer new improvements. Support is trying to bring these ideas to life, at the same time proposing their own changes. These tools and processes help us not only to support and quickly deliver software to productive, but also unite us as a DevOps team.
AS / 400 is used by many large Russian banks, and all of them are slowly moving towards CI / CD. Raiffeisenbank was one of the first (if not the first) to use these practices to install software on the AS / 400 platform in commercial operation.
Why did we come to this
We went a long way to automating installations; we were pushed for various reasons:
- The duration of the installation. Because of this, the product is for a long time unavailable to bank customers, and an additional burden falls on the night shift staff, although they are already busy with maintenance work.
- The quality of the installation. With the manual transfer of objects prepared for installation, something “suddenly” may disappear, or, on the contrary, appear. Backup of changeable objects is completely on the conscience of the duty personnel, and they can inadvertently install a new version of one of the package objects without saving the old one. What this leads to, we all know perfectly well.
After several unsuccessful installations, we decided to include a special installer in any installation on the AS / 400. For each specific delivery, the developers prepared a CL-script. He checked the completeness of the delivery, the availability of a backup of the objects being changed and the installation of the correct authorization on the objects, and also installed the software himself and handled abnormal situations.
')
The new approach was liked by the support service and the business, because the stability of the installations has improved. But it became more difficult for developers, because instead of a simple instruction like “take object xxx, move to another environment, put in yyy, add standard authorization” , I had to write an additional program for each installation.
Some problems remained unresolved. Tested software did not always correspond to what was put into productive. The developer could always change something in the prepared program at the last moment, guided by his vision of the correctness of the system. This could lead to disasters in the industrial exploitation environment.
In addition, it always took us a long time to move from stage to stage. After writing the code, the developer asked the tester: "Where to put something?"; after a while the tester answered and transferred the task to the developer; he set and sent the task back. Then “throwing the ball” began when correcting the found defects, while preparing the delivery for going into battle. At best, several minutes were spent on each such transition, at worst - days.
The support service was extremely important to understand for which task the productive objects were last changed, how to roll back the system to the previous version, in which non-industrial environments the change was made and what it was.
New AS / 400 Tools
With the emergence of CI / CD and DevOps movements in our bank, we received new tools and immediately began to master them.
The source code has moved to the appropriate repository — Git (Bitbucket), collected supplies — to Artifactory, and the build, installation, and testing process — to Bamboo.
If the RPG / C / CL source codes more or less smoothly fit into the code storage, because this is just text, then I had to tinker with the orchestrator of assemblies and installations (Bamboo), because standard Bamboo plug-ins for AS / 400 do not exist at the moment.
We began to try various installation options. The concepts were completely opposite, starting with “let's store all the installation and assembly logic in Bamboo as ftp / ssh scripts, without any additional code” , ending “but let's make a universal installer on AS / 400”.
As a result, we have a process that is practically no different from CI / CD processes for any other platform:
The central part of the process is Jira, it gathers all information about requirements, documentation, commits, assemblies and installations.
All the requirements appear in our form as Jira-tasks and get into the work of analysts who find out and describe the requirements and scenarios for using the software, after which the developers pull themselves to the task.
Developers write code and publish it in Bitbucket. The commit identifier and link to it automatically appears in the Jira task.
The commit in Bitbucket automatically launches the application build in Bamboo and the publication of the collected delivery in the Artifactory.
When the build is successfully completed, it launches a software installation plan in the developers' sandbox, where they can check what they have done. Also, after a successful installation, a set of autotests is launched, informing the developer about the quality of the new delivery. Information about the status of installations is immediately reflected in both the Jira task and Bitbucket.
If there are no defects, then the developer can proceed to the next task.
Then the tester is included in the process, who decides for himself where to install the testing software, and puts it by pressing a button.
After testing is completed, a “change request” is created for support professionals who validate and agree on the change. After that, the attendants at the right moment press the button for installation in Bamboo.
By the way, in one of the previous articles we wrote about our application, which is sensitive to long-term inaccessibility and processes millions of requests per day. It goes into the industrial environment only in the process described above.
Now I suggest a little immersed in the details of the assembly and installation on the AS / 400.
Assembly
Unfortunately, it is impossible to compile RPG / CL programs on * nix or Windows machines, so we had to complicate the standard software build scheme a bit:
For ourselves, we immediately identified the “reference” environment of AS / 400, on which Bamboo agents do a remote compilation.
The developer, creating new software, takes care of how it will be assembled and installed in the future. To do this, each software in Bamboo has its own Build and Deploy scripts.
After the commit, the Bamboo application build plan is automatically launched, configured for a specific repository.
The source code is downloaded and hosted on the build agent, and then FTP is put into a temporary build library on the AS / 400. Then ssh is compiled and the application build program is called. The task of the collector is to compile all the programs and the installer, and then put them in one package, ready for deployment in any environment.
Typical software package ready for installation:
Then the finished FTP package is transferred from AS / 400 to the build agent.
Here it should be noted that changing programs is only one part of the change, and the second is scripts for changing the database. They move from Bitbucket to the build agent, and join as another artifact to the already assembled package of AS / 400 programs.
After the build is complete, artifacts are published in Bamboo and Artifactory.
Bamboo's build plans are designed to be used for any AS / 400 software written in a bank. To add a new plan, we need 15 minutes - this takes into account the cloning of any other plan and changing the values of some variables.
Installation
One of the main advantages of our installation process is that it is repeatable and the same for any environment, be it a production environment or a development environment. We start testing the installation in the product already at the installation stage in the development environment.
In general, the installation looks like this:
By triggering or pressing a button, the deployment plan in Bamboo for the specified environment is launched. First of all, our artifacts are downloaded from the Artifactory to the installation agent - these are Liquibase-scripts and the installation package of programs (save-file).
Then, save file objects via FTP are moved and unpacked into a temporary library on the AS / 400, in an environment where the programs will be installed. Next, the Liquibase utility is launched, which, using the jt400 library, connects to the AS / 400 and executes the database update script, recording the corresponding entries in the change log.
Finally, Bamboo via ssh calls the installer, which backs up the objects to be changed, checks the contents of the delivery, sets the change and adjusts the authorization.
The installation result is sent to all team members subscribed to the list:
We added a small useful feature: the updated version of the software at the program level is assigned the task number, within which the last change was made, and the Bitbucket committee ID:
This helps to quickly find the latest change and see the changes in the code. In emergency situations, even the support service can make small changes to the code and send the fix to the product without having valuable knowledge about compiling, building and installing software.
We currently run about 150 installations in AS / 400 environments per day. We are constantly improving the process, it is not knocked out in stone and is not fixed in the standards and procedures - it lives and develops. Developers are coming to taste more and more, they have new ideas, they offer new improvements. Support is trying to bring these ideas to life, at the same time proposing their own changes. These tools and processes help us not only to support and quickly deliver software to productive, but also unite us as a DevOps team.
Source: https://habr.com/ru/post/341824/
All Articles