GitLab Shell Runner. Competitive launch of test services with Docker Compose

This article will be of interest to both testers and developers, but it is designed mostly for automators who are faced with the problem of setting up GitLab CI / CD for integration testing under conditions of insufficient infrastructure resources and / or lack of container orchestration platform. I will tell you how to configure the deployment of test environments using docker compose on a single GitLab shell runner so that when deploying several environments, the services that are started do not interfere with each other.

Content

Prerequisites

In my practice, it often happened to "heal" integration testing on projects. And often the first and most significant problem is the CI pipeline, in which integration testing of the service being developed (s) is performed in the dev / stage environment. This caused quite a few problems:
- Due to defects in a particular service, in the course of integration testing, the test loop may be corrupted by broken data. There were cases when sending a request with a broken JSON format hung the service, which caused the stand to be completely inoperable.
- The slowdown of the test circuit with the growth of test data. I think, to describe an example with cleaning / rollback of a DB does not make sense. In my practice, I have not met a project where this procedure would go smoothly.
- The risk of disrupting the performance of the test circuit when testing general system settings. For example, user / group / password / application policy.
- Test data from autotests interfere with live testers.
Someone will say that good auto tests should clean the data after themselves. I have arguments against:
- Dynamic stands are very convenient to use.
- Not every object can be removed from the system through the API. For example, the call to delete an object is not implemented, since it contradicts business logic.
- When creating an object through the API, a huge amount of metadata can be created, which is difficult to delete.
- If tests have dependencies among themselves, then the process of clearing data after performing tests turns into a headache.
- Additional (and, in my opinion, not justified) calls to the API.
- And the main argument: when the test data starts to clean directly from the database. It turns into a real PK / FK circus! From the developers you can hear: "I just added / deleted / renamed the nameplate, why did 100500 integration tests end up?"
In my opinion, the most optimal solution is a dynamic environment.
Many people use docker-compose to run a test environment, but very few people use docker-compose when performing integration testing in CI / CD. And here I do not take into account kubernetes, swarm and other container orchestration platforms. Not every company they are. It would be nice if docker-compose.yml were universal.
Even if we have our own QA runner, how can we ensure that the services launched via docker-compose do not interfere with each other?
How to collect logs of tested services?
How to clean a runner?

I have my own GitLab runner for my projects and I encountered these questions when developing a Java client for TestRail . Or rather, when you run the integration tests. Here we will solve these issues with examples from this project.

To the content

GitLab Shell Runner

For a runner, I recommend a Linux virtual machine with 4 vCPU, 4 GB RAM, 50 GB HDD.
On the Internet, a lot of information on how to configure gitlab-runner, so briefly:

Go to the machine for SSH
If you have less than 8 GB of RAM, then I recommend to make a swap 10 GB , so that the OOM killer does not come and do not kill us due to lack of RAM. This can happen when more than 5 tasks are running simultaneously. Tasks will be slower, but stable.
Example with OOM killer
If in the task logs you see bash: line 82: 26474 Killed , then simply execute sudo dmesg | grep 26474 sudo dmesg | grep 26474
```
 [26474] 1002 26474 1061935 123806 339 0 0 java Out of memory: Kill process 26474 (java) score 127 or sacrifice child Killed process 26474 (java) total-vm:4247740kB, anon-rss:495224kB, file-rss:0kB, shmem-rss:0kB 
```
And if the picture looks something like this, then either add a swap, or drop RAM.

Install gitlab-runner , docker , docker-compose , make.

Add user gitlab-runner to docker group

 sudo groupadd docker sudo usermod -aG docker gitlab-runner

Register gitlab-runner.
Open for editing /etc/gitlab-runner/config.toml and add
```
 concurrent=20 [[runners]] request_concurrency = 10 
```
This will allow to run parallel tasks on one runner. Read more here .
If your machine is more powerful, for example, 8 vCPU, 16 GB RAM, then these numbers can be made at least 2 times larger. But it all depends on what exactly will run on this runner and in what quantity.

It's enough.

To the content

Docker-compose.yml preparation

The main task is the docker-compose.yml, which will be used both locally and in the CI pipeline.

To run multiple instances of the environment, the COMPOSE_PROJECT_NAME variable will be used (see makefile ).

An example of my docker-compose.yml

 version: "3" #    web (php)  fmt , #      . #   ,   /var/www/testrail volumes: static-content: services: db: image: mysql:5.7.22 environment: MYSQL_HOST: db MYSQL_DATABASE: mydb MYSQL_ROOT_PASSWORD: 1234 SKIP_GRANT_TABLES: 1 SKIP_NETWORKING: 1 SERVICE_TAGS: dev SERVICE_NAME: mysql migration: image: registry.gitlab.com/touchbit/image/testrail/migration:latest links: - db depends_on: - db fpm: image: registry.gitlab.com/touchbit/image/testrail/fpm:latest container_name: "testrail-fpm-${CI_JOB_ID:-local}" volumes: - static-content:/var/www/testrail links: - db web: image: registry.gitlab.com/touchbit/image/testrail/web:latest #   TR_HTTP_PORT  TR_HTTPS_PORTS  , #     80  443  . ports: - ${TR_HTTP_PORT:-80}:80 - ${TR_HTTPS_PORT:-443}:443 volumes: - static-content:/var/www/testrail links: - db - fpm

To the content

Makefile Preparation

I use the Makefile, as it is very convenient for both local control of the environment, and in CI.

Further comments inline

 #           `.indirect`, #     `docker-compose.yml` #  bash   pipefail # pipefail -   ,      SHELL=/bin/bash -o pipefail #   CI_JOB_ID   ifeq ($(CI_JOB_ID),) #   local CI_JOB_ID := local endif #    export COMPOSE_PROJECT_NAME = $(CI_JOB_ID)-testrail #    , , volumes docker-down: docker-compose -f .indirect/docker-compose.yml down #   docker-down () docker-up: docker-down #     docker-registry docker-compose -f .indirect/docker-compose.yml pull #   # force-recreate -    # renew-anon-volumes -   volumes   docker-compose -f .indirect/docker-compose.yml up --force-recreate --renew-anon-volumes -d #  ,   ,           docker ps #    docker-logs: mkdir -p ./logs docker logs $${COMPOSE_PROJECT_NAME}_web_1 >& logs/testrail-web.log || true docker logs $${COMPOSE_PROJECT_NAME}_fpm_1 >& logs/testrail-fpm.log || true docker logs $${COMPOSE_PROJECT_NAME}_migration_1 >& logs/testrail-migration.log || true docker logs $${COMPOSE_PROJECT_NAME}_db_1 >& logs/testrail-mysql.log || true #   docker-clean: @echo   testrail- docker kill $$(docker ps --filter=name=testrail -q) || true @echo    docker rm -f $$(docker ps -a -f --filter=name=testrail status=exited -q) || true @echo  dangling  docker rmi -f $$(docker images -f "dangling=true" -q) || true @echo  testrail  docker rmi -f $$(docker images --filter=reference='registry.gitlab.com/touchbit/image/testrail/*' -q) || true @echo    volume docker volume rm -f $$(docker volume ls -q) || true @echo   testrail  docker network rm $(docker network ls --filter=name=testrail -q) || true docker ps

We check local start

 $ make docker-up docker-compose -f .indirect/docker-compose.yml pull Pulling db ... done Pulling migration ... done Pulling fpm ... done Pulling web ... done docker-compose -f .indirect/docker-compose.yml up --force-recreate --renew-anon-volumes -d Creating network "local-testrail_default" with the default driver Recreating local-testrail_db_1 ... done Recreating local-testrail_migration_1 ... done Recreating local-testrail_fpm_1 ... done Recreating local-testrail_web_1 ... done docker ps CONTAINER ID NAMES 3b8f9d4af29c local-testrail_web_1 5622c7d742d5 local-testrail_fpm_1 b580e3392038 local-testrail_migration_1 e467630bd3a5 local-testrail_db_1

Checking CI launch

 $ export CI_JOB_ID=123456789 $ make docker-up docker-compose -f .indirect/docker-compose.yml pull Pulling db ... done Pulling migration ... done Pulling fpm ... done Pulling web ... done docker-compose -f .indirect/docker-compose.yml up --force-recreate --renew-anon-volumes -d Creating network "123456789-testrail_default" with the default driver Creating volume "123456789-testrail_static-content" with default driver Creating 123456789-testrail_db_1 ... done Creating 123456789-testrail_fpm_1 ... done Creating 123456789-testrail_migration_1 ... done Creating 123456789-testrail_web_1 ... done docker ps CONTAINER ID NAMES ccf1ad33d0e8 123456789-testrail_web_1 bc079964f681 123456789-testrail_fpm_1 10dc9d4d8f2a 123456789-testrail_migration_1 fe98d43c380e 123456789-testrail_db_1

We check the collection of logs

 $ make docker-logs mkdir -p ./logs docker logs ${COMPOSE_PROJECT_NAME}_web_1 >& logs/testrail-web.log || true docker logs ${COMPOSE_PROJECT_NAME}_fpm_1 >& logs/testrail-fpm.log || true docker logs ${COMPOSE_PROJECT_NAME}_migration_1 >& logs/testrail-migration.log || true docker logs ${COMPOSE_PROJECT_NAME}_db_1 >& logs/testrail-mysql.log || true

To the content

Preparing .gitlab-ci.yml

Running integration tests

 Integration: stage: test tags: - my-shell-runner before_script: #   registry - docker login -u gitlab-ci-token -p ${CI_JOB_TOKEN} ${CI_REGISTRY} #   TR_HTTP_PORT  TR_HTTPS_PORT - export TR_HTTP_PORT=$(shuf -i10000-60000 -n1) - export TR_HTTPS_PORT=$(shuf -i10000-60000 -n1) script: #    - make docker-up #    jar (  ) - java -jar itest.jar --http-port ${TR_HTTP_PORT} --https-port ${TR_HTTPS_PORT} #    - docker run --network=testrail-network-${CI_JOB_ID:-local} --rm itest after_script: #   - make docker-logs #   - make docker-down artifacts: #   when: always paths: - logs expire_in: 30 days

As a result of launching such a task, the logs directory will contain logs of services and tests in artifacts. Which is very convenient in case of errors. In my case, each test in parallel writes its own log, but I will tell you about it separately.

To the content

Runner cleaning

The task will run only on schedule.

 stages: - clean - build - test Clean runner: stage: clean only: - schedules tags: - my-shell-runner script: - make docker-clean

Next, go to our GitLab project -> CI / CD -> Schedules -> New Schedule and add a new schedule

To the content

Result

Run 4 tasks in GitLab CI

In the logs of the last task with integration tests we see containers from different tasks.

 CONTAINER ID NAMES c6b76f9135ed 204645172-testrail-web_1 01d303262d8e 204645172-testrail-fpm_1 2cdab1edbf6a 204645172-testrail-migration_1 826aaf7c0a29 204645172-testrail-mysql_1 6dbb3fae0322 204645084-testrail-web_1 3540f8d448ce 204645084-testrail-fpm_1 70fea72aa10d 204645084-testrail-mysql_1 d8aa24b2892d 204644881-testrail-web_1 6d4ccd910fad 204644881-testrail-fpm_1 685d8023a3ec 204644881-testrail-mysql_1 1cdfc692003a 204644793-testrail-web_1 6f26dfb2683e 204644793-testrail-fpm_1 029e16b26201 204644793-testrail-mysql_1 c10443222ac6 204567103-testrail-web_1 04339229397e 204567103-testrail-fpm_1 6ae0accab28d 204567103-testrail-mysql_1 b66b60d79e43 204553690-testrail-web_1 033b1f46afa9 204553690-testrail-fpm_1 a8879c5ef941 204553690-testrail-mysql_1 069954ba6010 204553539-testrail-web_1 ed6b17d911a5 204553539-testrail-fpm_1 1a1eed057ea0 204553539-testrail-mysql_1

More detailed log

 $ docker login -u gitlab-ci-token -p ${CI_JOB_TOKEN} ${CI_REGISTRY} WARNING! Using --password via the CLI is insecure. Use --password-stdin. WARNING! Your password will be stored unencrypted in /home/gitlab-runner/.docker/config.json. Configure a credential helper to remove this warning. See https://docs.docker.com/engine/reference/commandline/login/#credentials-store Login Succeeded $ export TR_HTTP_PORT=$(shuf -i10000-60000 -n1) $ export TR_HTTPS_PORT=$(shuf -i10000-60000 -n1) $ mkdir ${CI_JOB_ID} $ cp .indirect/docker-compose.yml ${CI_JOB_ID}/docker-compose.yml $ make docker-up docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml kill docker network rm testrail-network-${CI_JOB_ID:-local} || true Error: No such network: testrail-network-204645172 docker network create testrail-network-${CI_JOB_ID:-local} 0a59552b4464b8ab484de6ae5054f3d5752902910bacb0a7b5eca698766d0331 docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml pull Pulling web ... done Pulling fpm ... done Pulling migration ... done Pulling db ... done docker-compose -f ${CI_JOB_ID:-.indirect}/docker-compose.yml up --force-recreate --renew-anon-volumes -d Creating volume "204645172-testrail_static-content" with default driver Creating 204645172-testrail-mysql_1 ... Creating 204645172-testrail-mysql_1 ... done Creating 204645172-testrail-migration_1 ... done Creating 204645172-testrail-fpm_1 ... done Creating 204645172-testrail-web_1 ... done docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES c6b76f9135ed registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" 13 seconds ago Up 1 second 0.0.0.0:51148->80/tcp, 0.0.0.0:25426->443/tcp 204645172-testrail-web_1 01d303262d8e registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" 16 seconds ago Up 13 seconds 9000/tcp 204645172-testrail-fpm_1 2cdab1edbf6a registry.gitlab.com/touchbit/image/testrail/migration:latest "docker-entrypoint.s…" 16 seconds ago Up 13 seconds 3306/tcp, 33060/tcp 204645172-testrail-migration_1 826aaf7c0a29 mysql:5.7.22 "docker-entrypoint.s…" 18 seconds ago Up 16 seconds 3306/tcp 204645172-testrail-mysql_1 6dbb3fae0322 registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" 36 seconds ago Up 22 seconds 0.0.0.0:44202->80/tcp, 0.0.0.0:20151->443/tcp 204645084-testrail-web_1 3540f8d448ce registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" 38 seconds ago Up 35 seconds 9000/tcp 204645084-testrail-fpm_1 70fea72aa10d mysql:5.7.22 "docker-entrypoint.s…" 40 seconds ago Up 37 seconds 3306/tcp 204645084-testrail-mysql_1 d8aa24b2892d registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" About a minute ago Up 53 seconds 0.0.0.0:31103->80/tcp, 0.0.0.0:43872->443/tcp 204644881-testrail-web_1 6d4ccd910fad registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" About a minute ago Up About a minute 9000/tcp 204644881-testrail-fpm_1 685d8023a3ec mysql:5.7.22 "docker-entrypoint.s…" About a minute ago Up About a minute 3306/tcp 204644881-testrail-mysql_1 1cdfc692003a registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" About a minute ago Up About a minute 0.0.0.0:44752->80/tcp, 0.0.0.0:23540->443/tcp 204644793-testrail-web_1 6f26dfb2683e registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" About a minute ago Up About a minute 9000/tcp 204644793-testrail-fpm_1 029e16b26201 mysql:5.7.22 "docker-entrypoint.s…" About a minute ago Up About a minute 3306/tcp 204644793-testrail-mysql_1 c10443222ac6 registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" 5 hours ago Up 5 hours 0.0.0.0:57123->80/tcp, 0.0.0.0:31657->443/tcp 204567103-testrail-web_1 04339229397e registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" 5 hours ago Up 5 hours 9000/tcp 204567103-testrail-fpm_1 6ae0accab28d mysql:5.7.22 "docker-entrypoint.s…" 5 hours ago Up 5 hours 3306/tcp 204567103-testrail-mysql_1 b66b60d79e43 registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" 5 hours ago Up 5 hours 0.0.0.0:56321->80/tcp, 0.0.0.0:58749->443/tcp 204553690-testrail-web_1 033b1f46afa9 registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" 5 hours ago Up 5 hours 9000/tcp 204553690-testrail-fpm_1 a8879c5ef941 mysql:5.7.22 "docker-entrypoint.s…" 5 hours ago Up 5 hours 3306/tcp 204553690-testrail-mysql_1 069954ba6010 registry.gitlab.com/touchbit/image/testrail/web:latest "nginx -g 'daemon of…" 5 hours ago Up 5 hours 0.0.0.0:32869->80/tcp, 0.0.0.0:16066->443/tcp 204553539-testrail-web_1 ed6b17d911a5 registry.gitlab.com/touchbit/image/testrail/fpm:latest "docker-php-entrypoi…" 5 hours ago Up 5 hours 9000/tcp 204553539-testrail-fpm_1 1a1eed057ea0 mysql:5.7.22 "docker-entrypoint.s…" 5 hours ago Up 5 hours 3306/tcp 204553539-testrail-mysql_1

All tasks completed successfully

Task artifacts contain logs of services and tests.

It seems everything is beautiful, but there is a nuance. Pipeline can be forcibly canceled during the execution of integration tests, in which case running containers will not be stopped. From time to time you need to clean the runner. Unfortunately, the revision task in GitLab CE is still in Open status.

But we added the launch of the task on a schedule, and no one forbids us to start it manually.
Go to our project -> CI / CD -> Schedules and run the task Clean runner