Building projects with GitLab CI: one .gitlab-ci.yml for hundreds of applications

The article solves the problem of managing the description of the assembly for a large number of similar applications. To make GitLab CI work in a project, you need to add the .gitlab-ci.yml file to the repository. But what if in a hundred repositories is a file with the same content? Even if you decompose it into repositories once, how then to change it? But what if one .gitlab-ci.yml not enough for building - Dockerfile or Dappfile , different scripts and structure of YAML files for Helm are needed? How to update them?



How to start solving the problem of building hundreds of similar applications? Of course, see if you can use GitLab CI to use .gitlab-ci.yml from another repository, or link .gitlab-ci.yml from files in other repositories ...

')

In search of this opportunity, the following issues immediately emerge:

• Add 'include' key to .gitlab-ci.yml ;
• Import / include common CI / CD content from one project into another's `.gitlab-ci.yml` (EE only?) ;
• Add includes capability in gitlab-ci YAML format .

It can be seen that the ability to have some kind of common .gitlab-ci.yml interests the community. The decision to add sections from a file in another repository seems very simple: it is based on many years of programming practice and will be clear to anyone. However, include how the concept works well in the case of the source tree, but in the case of several Git repositories in this solution, you can see the following disadvantages:

1. In include, you need to specify from which branch to take the file to connect, so the assembly will not be played.
2. In include, you need to specify from which branch to take the file for the connection, so you need to cache the effective .gitlab-ci.yml , store it and reassemble it already on the basis of it.
3. In some projects, it is necessary to solve clause 1, and in some - clause 2, but they are mutually exclusive.
4. If something has changed in the .gitlab-ci.yml file, then in fact the .gitlab-ci.yml project of the project that is going to change, but the change history will not be visible.

For the case with similar applications, two more minuses are added:

1. The problem with a hundred identical .gitlab-ci.yml remains.
2. The problem with updating additional files also remains.

A different angle

The include solution is a pull model, i.e. the project at assembly pulls out part of the CI configuration. If to replace pull with push, then it will turn out so:

• a common-ci-config project is created, which stores the common .gitlab-ci.yml and other files needed for building;
• creates a user gitlab-ci-distributor , who is given the right to push (Master) in the desired projects.

This option works as follows: the common-ci-config project .gitlab-ci.yml file common to hundreds of other projects. When this file is changed from the user gitlab-ci-distributor, commits are sent to other projects.



For the build files, you can choose: either add them to the commit, or to the .gitlab-ci.yml projects, add them to the build task of the common-ci-config project.



Advantages of this approach:

• In each project, it becomes visible when .gitlab-ci.yml and who changed it. The problem of storing the effective .gitlab-ci.yml , because Each project always shows the full version without include.
• In projects where the latest version of the assembly files is not needed at the time of assembly, the assembly files are added with a commit.
• In projects where the latest version is always needed at the time of assembly, the assembly files are cloned.
• It is possible to add a part of files to a commit, and a part - to use from a cloned copy.
• You can implement the concept of include for .gitlab-ci.yml by invoking a script. That is, if it is necessary that .gitlab-ci.yml always uses the latest version of the test configuration when building, then the testing is placed in a script in the common-ci-config project.

GitLab API

So, the problem is indicated and there is a solution. To continue, you need to talk about the GitLab API ( documentation on the GitLab website ). You will need the following methods:

• Commit file The API offers two options: modifying a single file and the designer of a commit with several files . The second option is a bit more complicated, but it has more features.
• Getting a list of projects . This method will help you know which projects need to be committed to.
• Information about the current commit . The method will be needed to copy the message of the commit and its author from the general repository to the rest.

API methods can be invoked using curl, and JSON, which is answered, can be processed using jq ( filter documentation ).



To call the methods, you need to create an access token. This will be further in the article, but for now - an example of how to get a list of projects in the group:

 $ curl -s --header "PRIVATE-TOKEN: $TOKEN" https://gitlab.example.com/api/v4/groups/group-of-alike-projects/projects?simple=true | \ jq -r '.[] | "\(.path_with_namespace)\t\(.id)"' group-of-alike-projects/project-pasiphae 7 group-of-alike-projects/project-megaclite 6 group-of-alike-projects/project-helike 5 group-of-alike-projects/project-erinome 4 group-of-alike-projects/project-callisto 3 group-of-alike-projects/project-aitne 2 group-of-alike-projects/project-adrastea 1 

GitLab Setup

Calling API methods is impossible without authorization. GitLab offers authorization through access tokens. To get such a token, you need to create a separate user who will be given the rights to manage the necessary repositories. Let it be the user of gitlab-ci-distributor :











Next you need to become this user and create access token:







To access projects where you need to manage assembly files, you need to add the user gitlab-ci-distributor to the group:







Project common files will be stored in the common-ci-config project. The project must be created in a separate group - for example, infra . In the project settings a secret variable is added with the value of the received token:







The described actions are performed by the administrator once. Further, all configuration is done through files in the common-ci-config repository.

Common-ci-config repository

Now you can test working with the API via GitLab CI. To do this, a simple .gitlab-ci.yml added to the common-ci-config .gitlab-ci.yml :

 stages: - distribute distribute: stage: distribute script: - ./distribute.sh 

... and the distribute.sh script, which so far displays information about the commit and projects from the selected group:

 #!/usr/bin/env bash curl -s --header "PRIVATE-TOKEN: $DISTRIBUTOR_TOKEN" https://gitlab.example.com/api/v4/projects/infra%2Fcommon-ci-config/repository/commits/$CI_COMMIT_SHA | jq '.' curl -s --header "PRIVATE-TOKEN: $DISTRIBUTOR_TOKEN" https://gitlab.example.com/api/v4/groups/group-of-alike-projects/projects?simple=true | \ jq -r '.[] | "\(.path_with_namespace)\t\(.id)"' 

The result of the distribute task:

 Running with gitlab-runner 10.1.0 (c1ecf97f) on gitlab (d82a6d8f) Using Shell executor... Running on gitlab... Fetching changes... HEAD is now at 08dcc92 Initial .gitlab-ci.yml and distribute.sh Checking out 08dcc92a as master... Skipping Git submodules setup $ ./distribute.sh { "id": "08dcc92abf0d951194ad1ffcc23deeb875855320", "short_id": "08dcc92a", "title": "Initial .gitlab-ci.yml and distribute.sh", "created_at": "2017-10-25T16:35:15.000+03:00", "parent_ids": [ "d9bdea91d081025c2af658209f23f684c96b5cee" ], "message": "Initial .gitlab-ci.yml and distribute.sh\n", "author_name": "root root", "author_email": "root.root@gitlab.example.com", "authored_date": "2017-10-25T16:35:15.000+03:00", "committer_name": "root root", "committer_email": "root.root@gitlab.example.com", "committed_date": "2017-10-25T16:35:15.000+03:00", "stats": { "additions": 0, "deletions": 0, "total": 0 }, "status": "running", "last_pipeline": { "id": 2, "sha": "08dcc92abf0d951194ad1ffcc23deeb875855320", "ref": "master", "status": "running" } } group-of-alike-projects/project-pasiphae 7 group-of-alike-projects/project-megaclite 6 group-of-alike-projects/project-helike 5 group-of-alike-projects/project-erinome 4 group-of-alike-projects/project-callisto 3 group-of-alike-projects/project-aitne 2 group-of-alike-projects/project-adrastea 1 Job succeeded 

Revision script distribute.sh

The script will distribute the common .gitalb-ci.yml . In order not to confuse it with the .gitlab-ci.yml project of the common-ci-config project, the file is located in the common directory. The file describes a simple automatic task:

 # common/.gitlab-ci.yml stages: - build build: stage: build script: - echo Building project $CI_PROJECT_PATH 

The distribute.sh script already has information about the commit and a list of projects. In order to get a beautiful commit into the projects, you need to select the author’s name and mail and the full message of the commit. You also need to add a cycle on the received projects and for each project, call the method that creates the commit.



Modified distribute.sh :

 #!/usr/bin/env bash COMMIT_INFO=$(curl -s --header "PRIVATE-TOKEN: $DISTRIBUTOR_TOKEN" https://gitlab.example.com/api/v4/projects/infra%2Fcommon-ci-config/repository/commits/$CI_COMMIT_SHA) #     ,  jq  -r MESSAGE=$(echo "$COMMIT_INFO" | jq '.message') AUTHOR_NAME=$(echo "$COMMIT_INFO" | jq -r '.author_name') AUTHOR_EMAIL=$(echo "$COMMIT_INFO" | jq -r '.author_email') CONTENT=$(base64 -w0 common/.gitlab-ci.yml) PAYLOAD=$(cat <<- JSON { "branch": "master", "commit_message": $MESSAGE, "author_name": "$AUTHOR_NAME", "author_email": "$AUTHOR_EMAIL", "actions": [ { "action": "update", "file_path": ".gitlab-ci.yml", "content": "$CONTENT", "encoding": "base64" } ] } JSON ) echo "$PAYLOAD" curl -s --header "PRIVATE-TOKEN: $DISTRIBUTOR_TOKEN" https://gitlab.example.com/api/v4/groups/group-of-alike-projects/projects?simple=true | \ jq -r '.[] | "\(.path_with_namespace)\t\(.id)"' | \ while read project do name=`echo $project | awk '{print $1}'` id=`echo $project | awk '{print $2}'` echo Update project $name curl -s --request POST --header "PRIVATE-TOKEN: $DISTRIBUTOR_TOKEN" \ --header "Content-Type: application/json" \ --data "$PAYLOAD" https://gitlab.example.com/api/v4/projects/$id/repository/commits done echo Stop 

The result of the distribute task:







In the group-of-alike-projects / project-pasiphae project, the commit will look like this:







The result of the build task in the group-of-alike-projects / project-pasiphae project :







You can see that the user who runs the task is gitlab-ci-distributor . But at the same time, the author of a commit is the user who made a commit in common-ci-config .

Disable simultaneous automatic build

The distribute.sh script adds commits to several projects at once. This leads to the creation of a new pipeline and the simultaneous launch of assembly orders. This effect is not always needed. To prevent a commit that updates .gitlab-ci.yml from running the build, you can first set up a task with a warning message:

 script: - 'if [ "x$GITLAB_USER_NAME" == "xgitlab-ci-distributor" ] ; then echo -e "\033[0;31m\n\n    .gitlab-ci.yml .\n\n\033[0m"; exit 1; fi' 

Attention! The variable GITLAB_USER_NAME appeared in GitLab 10.0 ( release of September 22, 2017) . In earlier versions there is only GITLAB_USER_ID and for the condition you will have to use the user ID. This ID can be found, for example, by running the task with script: [export] or with such an API request:

 curl -s --header "PRIVATE-TOKEN: $DISTRIBUTOR_TOKEN" https://gitlab.example.com/api/v4/users?username=gitlab-ci-distributor | jq '.[] | .id' 

Result:







If you run this task again, but from a regular user, everything will succeed:

Conclusion

In general, this information is enough to continue to experiment with mass project management.



For simplicity of experiments and repetition of what is described in the article, you can install GitLab in a virtual machine, for example, using the gitlab-vagrant project . Note that you have to fix Vagrantfile : change the base image to ubuntu/xenial64 and increase the memory of vb.memory = "3072" . And after launching add gitlab-runner according to the instructions



The following resources were used in developing the solution:

• Gitlab API ;
• Gitlab CI variables ;
• jq manual .

PS

Read also in our blog (and subscribe not to miss new publications!) :

• " GitLab CI for continuous integration and delivery in production. Part 1: our pipeline ";
• " GitLab CI for continuous integration and delivery in production. Part 2: Overcoming Difficulties . ”