Test server for the development team

Hi, Habr! In this article I want to share the experience of deploying a test server for the development team. In short, the essence of the problem - there is a development team and several projects in php. While we were few and the project was essentially one, then 1 test server was used and to show the task to the customer - the developer “started” the server for a certain time. If there were no “windows” in time, then you had to wait. Over time, the team grew and the complexity of the tasks increased, respectively, the test time and the employment of the test server increased, which negatively affected the deadlines and the bonus. Therefore it was necessary to search for the decision and it under a cat.

Introductory

What happened:

1. One test server
2. Gitlab and redmine on another server
3. The desire to understand the problem

All servers are in our local network, the test server is not accessible from the outside.

What was required:
')

1. Ability to test multiple projects / branches at the same time
2. The developer can go to the server, set it up and not break anything with others.
3. Everything should be as convenient as possible and do 1 button preferably from gitlab (CI / CD).

Solutions

1. One server, many hosts

The easiest option. We use the same test server, only the developer needs to create a host for each branch / project and make it into the nginx / apache2 configuration.

Pros:

1. Quick and clear to everyone
2. Can automate

Minuses:

1. Clause 2 of the requirements is not fulfilled - the developer can launch the update database and, under certain circumstances, put everything (Hi Andrey!)
2. Pretty complicated automation with a bunch of configuration files.

2. Each developer on the server!

Select each server and the developer himself is responsible for their own economy.

Pros:

1. The developer can fully customize the server for your project.

Minuses:

1. Clause 2 of the requirements is not fulfilled.
2. Expensive and resources can simply stand idle while development is underway, not testing.
3. Automation is even more complicated than in claim 1 due to different servers

3. Containerization - docker, kubernetes

This technology is increasingly penetrating our lives. At home I have been using docker for my projects for a long time.

Docker is an automation software for deploying and managing applications in a virtualization environment at the operating system level. Allows you to "pack" the application with all its environments and dependencies in a container that can be transferred to any Linux-system with cgroups support in the kernel, and also provides a container management environment.

Pros:

1. One server is used
2. All requirements are met.

Minuses:

1. Images and containers sometimes take up quite a lot of space, you have to crown clean already obsolete to free up space.

Docker implementation

When using gitlab, AutoDevOps, kubernetes settings were very often seen. Plus, bearded guys at various meetup tell how cool everything works with kubernetes. Therefore, it was decided to try to deploy the cluster at its facilities, the server was requested (and the test can not be touched, people are testing there) and rushed!

Since I have experience with kubernetes 0, everything has gone down on the manual with an attempt to understand how all these clusters work. After some time, I managed to raise the cluster, but then I went to problems with certificates, keys, and indeed with the difficulty of deployment. I needed a simpler solution to teach my colleagues to work with it (for example, I don’t want to spend the same vacation sitting on Skype and helping with the setting). Therefore, kubernetes was left alone. Docker itself remained and it was necessary to find a solution for container routing. Since they could be raised on different ports, it would be possible to use the same nginx for internal redirection. This is called a reverse proxy server.

Reverse proxy server - a type of proxy server that relays client requests from the external network to one or more servers that are logically located on the internal network. At the same time, it looks to the client as if the requested resources are located directly on the proxy server.

Reverse Proxy

In order not to reinvent the wheel, I began to look for ready-made solutions. And it was found - this is traefik .

Træfik is a modern HTTP reverse proxy and load balancer that simplifies the deployment of microservices. Træfik integrates with existing infrastructure components (Docker, Swarm mode, Kubernetes, Marathon, Consul, Etcd, Rancher, Amazon ECS, ...) and is configured automatically and dynamically. To work with docker, it is enough to specify its socket and everything, then Træfik finds all the containers and routing to them (for more details, see “Packing applications in docker”).


Here we inform the proxy that you need to listen to ports 80,443 and 8080 (web proxy), we mount the docker socket, the configuration file and the folder with certificates. For the convenience of naming test sites, we decided to make a local domain zone * .test. When accessing any site on it, the user gets to our test server. Therefore, the certificates in the traefik folder are self-signed, but it supports Let's Encrypt so much.

Certificate Generation

 openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout domain.key -out domain.crt 

Before starting, you need to create a proxy network in the docker (you can call it your own way).

 docker network create proxy 

This will be the network to link traefik with php container sites. Therefore, we specify it in the service’s networks parameter and in the entire file in networks, by specifying the external parameter: true.


It's all quite simple - we specify the http and https traffic entry points, do not forget to set insecureSkipVerify = true if the certificates are local. In the entryPoints.https.tls section, you can omit certificates, then traefik will substitute its certificate.

You can start the service

 docker-compose up -d 

If you go to the site.test address, you will get an error 404, since this domain is not tied to any container.

We pack applications in docker

Now you need to configure the container with the application, namely:

1. specify the network proxy network
2. add labels with traefik configuration

The following is the configuration of one of the applications.


In the app service, in the network section, you need to specify proxy and default, which means that it will be available in two networks, as you can see from the configuration, I do not forward ports to the outside, everything goes inside the network.

Next, configure the labels

  - traefik.enabled=true # traefik    - traefik.frontend.rule=Host:TEST_DOMAIN,crm.TEST_DOMAIN,bonus.TEST_DOMAIN #    traefik     - traefik.docker.network=proxy #   - traefik.port=443 #,     ssl   80   http - traefik.protocol=https #  #  phpmyadmin   http  

In the general section of networks, you need to specify external: true

The constant TEST_DOMAIN needs to be replaced with a domain, for example, site.test

Run the application

 docker-compose up -d 

Now if you go to the domains site.test, crm.site.test, bonus.site.test you can see the working site. And on the domain pma.site.test will be phpmyadmin for convenient work with the database.

GitLab Setup

Create a task handler, for this run

 gitlab-runner register 

Specify url gitlab, token and through which the task will be executed (executors). Since my test and gitlab are on different servers, I choose ssh executor. You will need to specify the server address and login / password to connect via ssh.

Runner can be attached to one or more projects. Since my work logic is the same everywhere, so a shared runner was created (common for all projects).
And the final touch is to create a CI configuration file.


This configuration describes 2 stages - build and clear. The build phase has 2 execution options - build_develop and build_prod



Gitlab builds a clear process flow diagram. In my example, all processes are started manually (when: manual parameter). This is done so that the developer, after deploying the test site, can pull his edits into the container without rebuilding the entire container. Another reason is the domain name - site $ CI_PIPELINE_ID.test, where CI_PIPELINE_ID is the number of the process that launched the build. That is, we gave the site with the site123.test domain for review and in order to make hot edits, changes to the container by the developer are immediately uploaded.

A small feature of the ssh executor. When you connect to the server, a view folder is created.

 /home//builds/_runner/0/_/_ 

Therefore, the line was added

 cd ../ && cp -r $CI_PROJECT_NAME $CI_PIPELINE_ID && cd $CI_PIPELINE_ID 

In it, we go up to the folder above and copy the project to the folder with the process number. So you can deploy multiple branches of one project. But in the settings of the handler you need to check Lock to current projects, so the handler will not try to deploy several branches at the same time.

The clear phase stops the containers and deletes the folder, you may need root privileges, so use the echo password | sudo -S rm, where password is your password.

Garbage collection

From time to time, it is necessary to remove unused containers in order not to take up space; for this, a script with such content is hanging in the crown

 #!/bin/bash #   : docker ps --filter status=dead --filter status=exited -aq | xargs -r docker rm -v #   : yes | docker container prune #    : yes | docker image prune #    : yes | docker volume prune 

performed once a day.

Conclusion

This solution helped us significantly optimize testing and the release of new features. Ready to answer questions, constructive criticism is accepted.

Bonus

In order not to collect images from the Dockerfile each time, you can store them in the local docker registry.


In this embodiment, authentication is not used, it is not a secure method (!!!), but it is suitable for storing non-critical images.

You can configure gitlab to view

  gitlab_rails['registry_enabled'] = true gitlab_rails['registry_host'] = "registry.test" gitlab_rails['registry_port'] = "5000" 

After that, a list of images appears in gitlab.