The story of how we accelerated tests 12 times

Accelerate the tests, they said.

And now, almost half a year has passed, as we rewrote our old un-healed, long-term and unstable functional tests and switched to quick, component-independent ones. Therefore, it's time to share :)

For those who do not know, component tests are tests that are completely isolated from the global environment and allow you to test certain cases that the unit test could not cover.
')
Half a year ago, the release of any feature, happened to take more than an hour, taking into account the fact that the code has long been on the master and fully tested, but the master branch can’t achieve a green bamboo assembly and then the question was how to live?

After all, in this case there is more harm than goodness to the tests, but getting rid of it and “hammering” on the tests is not the best option :) Then, a small micro-team was formed with the team lead:

1. Timlida
2. Developer backend
3. Engineer qa
4. Admina

Having quickly collaborated, we shared our tasks and one of them was to set up the environment for writing component tests. This is how my journey began.

At the time of the development, we had 140+ functional tests that were run in several streams for different environments (Frontend, Mobile, Backend) and they passed ~ 5-7 minutes; also often had to restart them in order to achieve a green assembly. And these tests fell no longer because of the new written code, but because of problems in the environment, that is, somewhere the API did not respond, somewhere the test microservice fell, etc. This stopped the work of the entire department, since the assemblies were launched almost every 5-10 minutes: someone rebuilt, someone was pushing a new code ...

After the first half of the week, we came to the conclusion that we would be “mopping” our API and third-party services, which would give us a completely isolated testing environment. But the question arose: to write something of your own or ... So, on this “or” everything ended - with a brief search, I met on my way - a small experience in the form of the “http-api-mock” Mock server.

http-api-mock is a lightweight and easy-to-install mock server written in Go with good documentation.

After hundreds of attempts to launch, as well as a general insight into the topic of mocks, I still managed to rewrite 1 functional test, which created a new ad on the site and, having gone through all the circles of hell, was convinced that the title on the page corresponds to the title in the object's body.
Imagine earned! The rewritten test turned out to be 3 times faster than the previous one, since here we did not check the creation, moderation, but immediately gave the desired object from the mock and won on it. This small victory was a good incentive for further development of this topic, thus, a week later, we had a new suite in the codeception called “component”, which already had the basic helper class for working with our Mock server and was launched at that time me on the sandbox

The base assistant class is able to create an ad as a json file in the configs directory of our mock server, give the desired ad by id, etc. Almost API.

The rest of the magic was waiting for us further - now it was necessary to set up a plan of assemblies in bamboo. So that our tests now pass through our CI & CD.

The admin helped us in this with setting up the launch of all these tests in the docker, which allowed us to raise our container with the Mock server for each build and run our tests in a completely isolated environment, without deploying our code to any test backend, which also could not Do not speed up the passage of our tests.

For all this magic to work, we had to add new config files with a new API address and external services, as well as raise a copy of the mysql database, and also create a new task in the build plan with the launch of our mock server.

# Delete/Create network docker network rm mock-kolesa-net; docker network create --subnet=IP_ADDR/24 --gateway IP_ADDR_GATEWAY mock-kolesa-net; # Docker run http-mock-kolesa docker run \ --rm \ --name http-mock-kolesa -d \ -v ${CONFIG}/config/:/config \ -v ${CONFIG}/data/:/data \ --user $(id -u):$(id -g) \ --net mock-kolesa-net \ --ip IP_ADDR\ local-docker-hub.kolesa-domain.org:7979/build/http-mock-kolesa; 

Now for our code there is a completely new API, which, regardless of any problems with the environment, will give us what we want.

As time went on, the tests corresponded and here 140 functional tests turned into 103 component tests that pass in parallel in ~ 30 seconds.

Of the benefits

Very nimble . Due to the fact that they are completely independent of the test environment, they do not have to go for data somewhere far away.

Stable . Tests do not have to worry if our API or any other service has dropped there, so we are always confident that the result will come to us.

Easy to write . Actually, in the process of rewriting, much was solved by copying the code from the old functional test to the new component and preparing for it endpoints in the Mock server.

Of the minuses

Support Now each developer who has made changes to the returned answer for a specific endpoint in the API also goes to the repository with configs for the mock server and rules the answer there.

A bunch of files . We decided to store the data with the configs as files, that is, each answer for the endpoint lies as a file and can be lost somewhere.

Results:

Tests
It was: 140+ functional tests = 5-7 minutes.
It became: 103 component tests = ~ 30 seconds.

Build stability
It was: Every third build fell due to any problems.
It became: It falls only when the developer broke the logic of some method.

In future, we have to rewrite acceptance (gui) tests - also run them inside the container and isolate them from the rest of the environment.