Geekly Articles each Day
📜 ⬆️ ⬇️

GitLab for Continuous Delivery project on InterSystems technology

In this article I would like to tell you about the organization of Continuous Integration / Continuous Delivery processes that automate the assembly, testing and delivery of applications on InterSystems platforms.


Consider topics such as:



Git 101


Despite the fact that the main topic is CD, Git , or rather a number of fundamental principles underlying it, had a great influence on software development processes, so we start with a brief description of the basic terms of Git.


Git is a version control and management system based on ideas such as:



Git was not the first version control system in which these ideas were implemented, however Git popularized these ideas and greatly simplified their practical application.


Repository (Repository)


The place where any data is stored and maintained.



Local repository - a repository located on the local computer
Remote repository - a repository located on a remote server


Commit


Recorded state of the repository.
Stores the difference (Diff) from any other commit called parent.


Parents can be:



Branch (Branch)


Pointer to a commit.
You can always look at his story before the first commit. For example, the master branch:


Branch


Commit tree


Commits form trees - a graphical representation of the repository. Consider the simplest linear option when three more commits were added to the two existing commits from the image above:


branch


Now the tree example is more complicated: two developers work at the same time and, in order not to interfere with each other, each of them works in his own branch. It looks like this:


branch


After a while, they need to merge the changes; for this, there are merge requests — requests to merge two states of the repository into one new state. In our case, the request to merge the develop branch into the master branch. After the request has been reviewed and approved and the merge occurred, the repository looks like this:


branch


After that, the development continues:


branch


Conclusions - Git 101



Development Methodologies (Git flows)


Git-based development methodology is a series of software development approaches using Git as the basis for development. There are many development methodologies, consider two of them:



Github flow


GitHub flow is probably one of the easiest Git based development methodologies. Here she is:



In addition, there are a number of rules:



The environment is the configured resource where your application code is executed. It can be a server, a virtual machine, or even a container.


Here's what it looks like:


Github


In detail about GitHub flow on Habré already written more than once .


Gitlab flow


If you are not ready to automatically update the code on an industrial environment, GitLab flow provides a unification of GitHub flow with several environments. Here's how it works: development is conducted in the same way as GitHub flow — in separate branches, which also merge into master , but the contents of the master branch are deployed on a test server. Additionally, you have branches of environments whose contents match the contents of your environments. There are usually three environments, but they can be more or less depending on your requirements:



The development process looks like this:



In detail about GitLab flow on Habré too wrote .


Conclusions - Development Methodologies (Git flows)


There are a number of development methodologies based on Git - from simple to complex. Choose a methodology that is not over complicated on the one hand, but on the other provides an adequate level of control over the status of your project.


Gitlab workflow


GitLab Workflow is a software development methodology that covers not only the development stages but the entire product life cycle from idea to user feedback. Here is what it looks like:




Each stage is described in detail on the GitLab website ; I will limit myself to a description of several stages.


Tasks and planning


The initial stages of GitLab Workflow are focused on the task — a new functionality, an error, or some other specific amount of work. The task has several goals, such as:



The planning phase allows you to group tasks according to their priority, stage, status.



We discussed the development above, follow any git development methodology. After we developed our new functionality and merged it into the master branch - what next?


Continuous Delivery (Continuous Delivery)


Continuous Delivery is a software development approach in which teams develop software in short sprints, ensuring that a new version of the application can be released at any time. This approach automates the assembly, testing and delivery of software. This approach helps to reduce the costs and risks of making changes, allowing you to get fast incremental updates for industrial applications. For continuous delivery, it is important to set up a simple and reproducible delivery process.


Continuous Delivery to GitLab


In GitLab, continuous delivery configurations are defined separately for each repository and stored in the YAML configuration file in the repository root.



Script


Defines one action and what conditions must be met to trigger it:



The environment is a configured server or container in which you can run scripts.


Runner is a service that runs scripts in a specific environment. Connected to GitLab and execute scripts as needed.


Runner can be deployed on a server, in a container, or even on a local development computer.


How is the continuous delivery?



Here is an example of a launch:



It consists of four stages performed sequentially.



As we see, each script was executed successfully, if one of the scripts failed, the following scripts would not be executed:



If you open the script, you can see why it ended with an error:


Script execution log
 Running with gitlab-runner 10.4.0 (857480b6) on test runner (ab34a8c5) Using Shell executor... Running on gitlab-test... Fetching changes... Removing diff.xml Removing full.xml Removing index.html Removing tests.html HEAD is now at a5bf3e8 Merge branch '4-versiya-1-0' into 'master' From http://gitlab.eduard.win/test/testProject * [new branch] 5-versiya-1-1 -> origin/5-versiya-1-1 a5bf3e8..442a4db master -> origin/master d28295a..42a10aa preprod -> origin/preprod 3ac4b21..7edf7f4 prod -> origin/prod Checking out 442a4db1 as master... Skipping Git submodules setup $ csession ensemble "##class(isc.git.GitLab).loadDiff()" [2018-03-06 13:58:19.188] Importing dir /home/gitlab-runner/builds/ab34a8c5/0/test/testProject/ [2018-03-06 13:58:19.188] Loading diff between a5bf3e8596d842c5cc3da7819409ed81e62c31e3 and 442a4db170aa58f2129e5889a4bb79261aa0cad0 [2018-03-06 13:58:19.192] Variable modified var=$lb("MyApp/Info.cls") Load started on 03/06/2018 13:58:19 Loading file /home/gitlab-runner/builds/ab34a8c5/0/test/testProject/MyApp/Info.cls as udl Load finished successfully. [2018-03-06 13:58:19.241] Variable items var="MyApp.Info.cls" var("MyApp.Info.cls")="" Compilation started on 03/06/2018 13:58:19 with qualifiers 'cuk /checkuptodate=expandedonly' Compiling class MyApp.Info Compiling routine MyApp.Info.1 ERROR: MyApp.Info.cls(version+2) #1003: Expected space : '}' : Offset:14 [zversion+1^MyApp.Info.1] TEXT: quit, "1.0" } Detected 1 errors during compilation in 0.010s. [2018-03-06 13:58:19.252] ERROR #5475: Error compiling routine: MyApp.Info.1. Errors: ERROR: MyApp.Info.cls(version+2) #1003: Expected space : '}' : Offset:14 [zversion+1^MyApp.Info.1] > ERROR #5030: An error occurred while compiling class 'MyApp.Info' ERROR: Job failed: exit status 1

A compilation error caused the script to fail.


Let's move from theory to practice.


Install GitLab


We will install GitLab on our own server. However, you can use GitLab.com. There are many ways to install GitLab: from source, from a package, in a container. Choose the one you like and follow the installation instructions .


Requirements:



Configuration


First of all, set up email notifications .


Next, I recommend installing Pages. As mentioned above, artifacts from running scripts can be uploaded to GitLab. Users can download them, but it is often useful to open them directly in the browser and for this you need to install the pages.


Why do we need pages:



Since automatic redirection can be added to the HTML when the page is loaded, you can direct the user to the page with the results of unit tests.


HTML Generation Code
 ClassMethod writeTestHTML() { set text = ##class(%Dictionary.XDataDefinition).IDKEYOpen($classname(), "html").Data.Read() set text = $replace(text, "!!!", ..getURL()) set file = ##class(%Stream.FileCharacter).%New() set name = "tests.html" do file.LinkToFile(name) do file.Write(text) quit file.%Save() } ClassMethod getURL() { set url = "http://host:57772" set url = url _ $system.CSP.GetDefaultApp("%SYS") set url = url_"/%25UnitTest.Portal.Indices.cls?Index="_ $g(^UnitTest.Result, 1) _ "&$NAMESPACE=" _ $zconvert($namespace,"O","URL") quit url } XData html { <html lang="en-US"> <head> <meta charset="UTF-8"/> <meta http-equiv="refresh" content="0; url=!!!"/> <script type="text/javascript">window.location.href = "!!!"</script> </head> <body> If you are not redirected automatically, follow this <a href='!!!'>link to tests</a>. </body> </html> }

By the way, I ran into a bug when using pages (Error 502 when viewing artifacts) - this is the solution .


Connecting Environments to GitLab


Environments are required to run CD scripts — configured servers to run your application. To begin with, assume that you have a Linux server installed with the InterSystems platform (say, InterSystems IRIS, but everything will work with Caché or Ensemble). To connect the environment with GitLab you need:


  1. Install GitLab runner .
  2. Register the GitLab runner with GitLab.
  3. Allow gitlab-runner user to call InterSystems IRIS.

Important note on installing GitLab runner - DO NOT clone the server after installing GitLab runner. Results are unpredictable and undesirable.
I'll tell you more about steps 2 and 3.


Register the GitLab runner with GitLab.


After running the command: sudo gitlab-runner register


Several options will be offered to choose from, and although most of the steps are fairly simple, some of them are worth commenting on:



Several tokens are available: for the entire system (available in the administration settings) or for one project (available in the project settings).
When you connect the runner for the CD of a specific project, you need to specify the token of this particular project.



In the CD configuration, you can filter which scripts are executed for environments with specific tags. Therefore, in the simplest case, specify one tag, which will be the name of the environment.



Regardless of whether you use docker, choose a shell to run scripts.


Allow gitlab-runner user to call InterSystems IRIS.


After connecting to GitLab, you need to allow the gitlab-runner user to call InterSystems IRIS. For this:


  1. The gitlab-runner user must have rights to call irissession or csession . To do this, add it to the irisusr group or cacheusr command: usermod -a -G cacheusr gitlab-runner
  2. In InterSystems IRIS, create a gitlab-runner user and grant it permissions to execute CD scripts (write to the database, etc.)
  3. Enable OS authentication .

Instead of points 2 and 3, you can use other approaches, such as transferring a user / password, but the option with OS authentication seems to me more preferable.


CD configuration


So, let's start writing the configuration of continuous delivery. First, we describe the environment and the plan.


Environments


We have several environments and their corresponding branches:


EnvironmentBranchDeliveryWho can commitWho can drain
TestmasterAutomaticDevelopers, OwnersDevelopers, Owners
ExperiencedpreprodAutomaticNo oneOwners
IndustrialprodBy pressing a buttonNo oneOwners

Work plan


  1. Task setting, development and automated testing
    • The owner sets the task
    • The developer creates a feature-branch and commits the code to the problem solving there.
    • Developer merges feature branch into master
    • The new code is automatically delivered to the test environment and tested.
  2. Delivery to an experienced environment
    • The developer creates a request to merge master code into preprod
    • Owner approves of merging code from master into preprod
    • New code is automatically delivered to the expert environment.
  3. Delivery to the industrial environment
    • The developer (or owner) creates a request to merge code from preprod into prod
    • Owner approves of merging preprod code into prod
    • The owner clicks the "Expand" button
    • The new version of the application takes place in the automatic environment.

The same in graphic form:



application


Our test application consists of two parts:



Stages


From the above plan, we can distinguish the steps that we need to define in our continuous delivery configuration:



Let's start to make a configuration in the .gitlab-ci.yml :


 stages: - load - test - package - deploy

Scripts


The next part of the configuration is scripts. Documentation


Load


Let's start with the load server script that loads the server code.


 load server: environment: name: test url: http://test.hostname.com only: - master tags: - test stage: load

What's going on here?



Now you need to create the isc.git.GitLab class. All entry points to it should look like this:


 ClassMethod method() { try { // code halt } catch ex { write !,$System.Status.GetErrorText(ex.AsStatus()),! do $system.Process.Terminate(, 1) } }

This method can only be completed in two ways.



Here is the download code:


 /// Do a full load /// do ##class(isc.git.GitLab).load() ClassMethod load() { try { set dir = ..getDir() do ..log("Importing dir " _ dir) do $system.OBJ.ImportDir(dir, ..getExtWildcard(), "c", .errors, 1) throw:$get(errors,0)'=0 ##class(%Exception.General).%New("Load error") halt } catch ex { write !,$System.Status.GetErrorText(ex.AsStatus()),! do $system.Process.Terminate(, 1) } }

This method calls two other methods:



But how can we get a directory with a repository?


When GitLab executes a script, it defines a series of environment variables . One of them is CI_PROJECT_DIR - the full path to the repository root. So you can get it in the getDir method:


 ClassMethod getDir() [ CodeMode = expression ] { ##class(%File).NormalizeDirectory($system.Util.GetEnviron("CI_PROJECT_DIR")) }

Test


Here is the test run script:


 tests: environment: name: test url: http://test.hostname.com only: - master tags: - test stage: test script: csession IRIS "##class(isc.git.GitLab).test()" artifacts: paths: - tests.html

What changed? Of course, the name and code of the script, but still added artifacts . An artifact is a collection of files and folders that are attached to a script after it has been successfully completed. In our case, after completing the tests, we can generate an HTML page redirecting the user to the test results.


Note the similarity of the test scripts and downloads. Parts of scripts, such as environments, the same in all scripts can be separated into separate blocks. Define the test environment:


 .env_test: &env_test environment: name: test url: http://test.hostname.com only: - master tags: - test

Now our tests script looks like this:


 tests: <<: *env_test script: csession IRIS "##class(isc.git.GitLab).test()" artifacts: paths: - tests.html

Now we will write the corresponding server code that calls the unit tests (the article on the habr ):


 /// do ##class(isc.git.GitLab).test() ClassMethod test() { try { set tests = ##class(isc.git.Settings).getSetting("tests") if (tests'="") { set dir = ..getDir() set ^UnitTestRoot = dir $$$TOE(sc, ##class(%UnitTest.Manager).RunTest(tests, "/nodelete")) $$$TOE(sc, ..writeTestHTML()) throw:'..isLastTestOk() ##class(%Exception.General).%New("Tests error") } halt } catch ex { do ..logException(ex) do $system.Process.Terminate(, 1) } }

tests — . , .


writeTestHTML ( ) - -.


Package


-, REST API:


 <html> <head> <script type="text/javascript"> function initializePage() { var xhr = new XMLHttpRequest(); var url = "${CI_ENVIRONMENT_URL}:57772/MyApp/version"; xhr.open("GET", url, true); xhr.send(); xhr.onloadend = function (data) { document.getElementById("version").innerHTML = "Version: " + this.response; }; var xhr = new XMLHttpRequest(); var url = "${CI_ENVIRONMENT_URL}:57772/MyApp/author"; xhr.open("GET", url, true); xhr.send(); xhr.onloadend = function (data) { document.getElementById("author").innerHTML = "Author: " + this.response; }; } </script> </head> <body onload="initializePage()"> <div id = "version"></div> <div id = "author"></div> </body> </html>

"", ${CI_ENVIRONMENT_URL} . (npm), . :


 package client: <<: *env_test stage: package script: envsubst < client/index.html > index.html artifacts: paths: - index.html

Deploy


, index.html -.


 deploy client: <<: *env_test stage: deploy script: cp -f index.html /var/www/html/index.html

That's all!



? , . :


 stages: - load - test .env_test: &env_test environment: name: test url: http://test.hostname.com only: - master tags: - test .env_preprod: &env_preprod environment: name: preprod url: http://preprod.hostname.com only: - preprod tags: - preprod .script_load: &script_load stage: load script: csession IRIS "##class(isc.git.GitLab).loadDiff()" load test: <<: *env_test <<: *script_load load preprod: <<: *env_preprod <<: *script_load

.


CD , .


findings


— , , , . , . , .


InterSytems, InterSystems IRIS Data Platform ( Caché Ensemble), , , , .


Links



')

Source: https://habr.com/ru/post/354158/

More articles:


All Articles