Configure ClickHouse for integration testing in gitlab-ci

We had a service on golang, a separate kafka topic, clickhouse, gitlab-ci and a falling pipeline, a rotten ssh-key, and that’s all, and another holiday season, terrible downpours in the city, a broken laptop, alerts at night, and burning products . Not that all this was necessary for this article, but once you show the typical everyday life of a tester, then go in your intention to the end. The only thing that bothered me was p0. In the world there is nothing more desperate, gloomy and depressed than the tester, who missed it on the prod. But I knew that pretty soon I would plunge into it.

Why all this?

There is a common bunch of services — the service itself that does something — and the database into which these results are recorded. sometimes it happens directly, that is, “service - base”. In my case, the recording takes place through an intermediary, that is, “service - turn - base”.

So, there are several elements, and the border of these elements is the output of one and the input of the other - this is the place where problems arise. They just can not appear there.

A vivid example: in the service, the price field is processed as float32, in the database it is configured as decimal (18, 5), we send the maximum value of float32 as a test case from the output of the service to the base - oh, the base is not responding. Or already a more sad example - the base does not crash, but in the logs there are no data writing errors in the database. just the database stops piling up. Or the record passes, but with data loss or with distortion: the field leaves the service as float64, and is written as float32.
')
Or, during the service life cycle, it was decided that it was necessary to change the type of a particular field. The field has long been implemented for sale, but now you need to edit it. And of course they changed it only in one place. Khoba, something went wrong again.

Task

I do not want to follow all these changes. I want it to not fall. I want the recording to take place correctly.

Output: integration tests!

Implementation and difficulties

Where to break?

There is a dev-environment: terribly unstable and usually used by developers as a sandbox. It creates chaos and anarchy, which are characteristic of a hard backend.

There is a test environment or a qa-booth: it’s already set up better, even devops are watching it, but if you don’t kick them, nothing will happen. and this environment is updated frequently. and more often there is something broken.

And there is a food - the holy of holies: it is better not to chase anything like that on it. integration tests suggest the possibility of a bug that they must find before it gets on the prod.

So what to do with the environment when it is either unstable or military? That's right, create your own!

What to do with the base?

The base can be run in several ways.

As we have already discussed above, we will not connect to the real base of this or that environment.

First, you can raise the clickhouse -server with the necessary settings, roll out the required sql on it and communicate with it through the clickhouse-client. At the very first successful attempt to put a similar base, sad and ci. tests zafeylilis, the server did not go out and continued to work. Let's just say, it’s still a mystery to me why it started at all. (it is itself, I have nothing to do with it). I do not recommend this option.

A convenient option out of the box is the use of a docker image .
Download the version to your machine. Clickhouse to start need config.xml with settings. More here
For the clickable image to be reused, you need to create the correct dockerfile. We indicate in it that we want to copy config.xl to a folder, we drop other required configs. Be sure to copy the scripts to deploy your base.

Since we will turn to the image from the outside, we need to open those ports through which we will communicate with the klykhaus. The click works for 8123 over http and 9000 over tcp.

We get the following dockerfile:

From yandex/clickhouse-server Expose 8123 Expose 9000 Add config.xml /etc/clickhouse-server/config.xml Add my_init_script.sql /docker-entrypoint-initdb.d/ 

How to throw an image in ci?

In order to somehow work in ci with a docker-image, it must be called there somehow.

You can commit and launch an image into your repository and, as part of running the tests, run the docker run with the necessary parameters. Only here the docker-click image weighs under 350mb. it is indecent to keep such files in git.

In addition, if the same docker image is needed on different projects (for example, different services are written in the same database), then all the more you should not do this. You can use the image storage docker registry
We believe that in our project it already exists and is being used. Therefore, log in, collect the docker-image and push it there.

 docker build -t my_clickhouse_image . docker login my_registry_path.domain.com docker push my_clickhouse_image 

Out and our image flew into the registry. Be sure to specify the tag when assembling!

The base is ready.

Read more about the registry here.

What to do with ci?

How in one step to start and your service, and the base?

It all depends on how we start and use the service. If you work with the service as with a docker-image, and indeed the whole .gitlab-ci.yml only works with them, then everything is simple.
There is a stray dind - docker-in-docker . It is indicated as the main service with which ci works, and allows the docker to use fully, and not to strain at all.

We pump out the latest image, add a stage of the required testing in stages and describe our sequence of actions.

 image: docker:stable services: - docker:dind stages: - build … - test-click ... - test - release … test-click: variables: VERY_IMPORTANT_VARIABLE: “its value” before_script: - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY script: - docker pull My_Service_Image - docker pull My_ClickHouse_Image - docker run -FLAGS My_ClickHouse_Image - docker run My_Service_Image /path/to/tests 

The official docker docker states that it is not recommended to use dind , but if you really need ...

In my project, the service should be tested through the launch of the binary. Here the magic begins
To do this, use the database as a service. The official gitlab-ci documentation lists the use of a container with a base as an example of the most common use case for a docker container in ci. Even examples of the settings mysql, postress and redis are given. But we are not looking for easy ways, we need clickhouse.

Connect the base! Be sure to specify alias. if it is not specified, a random name and random ip will be assigned to the database. That is, it will be unclear exactly how to apply to it. With alias, there will be no such problem - in the code of tests, the call will look like, for example, at http://my_alias_name:8123 .

For tests, the image of the base is also required, which we diligently started in the registry. to download the image, you need to perform docker login and docker pull, only ci does not know what a docker is - you need to install it.

The resulting code for the step in gitlab-ci.yml is:

 Integration tests: Services: - name: my_clickhouse:latest alias: clicktest Stage: tests Variables: Variables_for_my_service: “value” Before_script: - curl -ssl https://get.docker.com/ | sh - docker login -u gitlab-ci-token -p $ci_build_token my_registry_path.domain.com Script: - ./bin/my_service & - go test -v ./tests -tags=integration Dependencies: - build 

Profit

• I have a working bunch of service bazka.
• As part of the autotest, it is easy to access the database - just by alias.
• I reset the entries and settings of the database as part of the setup test, I call the service work, he writes to the database, I appeal to the database, I see that the database has not fallen off, I look at what has come, I validate. throwing more tests.
• You can not test the pens!

results

It would seem that a couple of lines of customization in gitlab-ci. Assembling a docker image is easy. Starting a bazka locally is easy. I got integration with the first tests that found problems in a day. But attempts to launch it into ci turned into a week of pain and hopelessness. And now, in the weeks of pain and hopelessness of the developers, who will have to fix everything they have programmed.

What have we managed to do?

• We set up a container with clickhouse.
• Push the container into the local storage.
• Learned to pull this image in step ci.
• Launched it in the runner.

Easily sent data to the database and accessed it from the test.

Automation is a fairly simple way to rid yourself of the routine of manual piercing integration.

What is important to pay attention to: make sure that the input base types correspond to the output types of the previous link. (and documentation , if any ).