Three years of autotests: how to increase speed and not only

Hi, I'm Alexey, a full-stack developer of the Vimbox platform. When I came to Skyeng, they decided here whether it was worth the time to spend on the autotest system and asked me to share my experience from the previous work. And I had such experience: by the time of leaving the previous place we wrote in php and twisted more than 3 thousand tests. As a result, I made a small internal presentation telling about the rake that I managed to tread in a few years of developing these auto-tests, struggling for their speed, code readability and overall efficiency. The presentation seemed to colleagues useful, so I put it in the text to be useful also to a wider audience.

To begin with, the terms that will be discussed in the article:

• Acceptance test - end-to-end test: here the browser or browser emulator executes the script
• Unit test (unit test) - method test
• Functional test - a test of a controller or component, when it comes to frontend
• Fixture - the state of the test environment necessary for the test to work (global variables, data in the database and other participants in the test script)

Pros and cons of different types of tests

Acceptance tests

• Pros: obvious from the name, such tests cover the entire system from top to bottom, make sure that everything works as it should.
• Cons: the feedback from these tests is very slow, they work for a long time, they are not very reliable, there are a lot of false positives. At a previous job, we were also faced with the fact that web drivers did not detect some elements that we saw with our eyes. Now this has probably been fixed, but then I had to abandon them.

Unit tests

• Pros: easy to write, work quickly. They cover a small piece of code, you do not need many states, therefore, you do not need a large fixture either.
• Cons: unstable to changes in the architecture or internal structure of the code. If you need to merge two methods into one or separate, select a class, delete a method, you have to rewrite the tests.

Functional tests are an intermediate solution.

• Pros: more reliable acceptance, more resistant to changes in the structure of the code than modular.
• Cons: slower than modular, more difficult to write, because need to prepare a big fixture.

The fight for speed

At the old work, we wrote a lot of functional tests, and the main challenge was the response speed. I had to wait a long time for the result, even with a local launch on the developer's computer. The speed was so low that it was not possible to apply the “development through testing” approach, since it involves running autotests several times per hour. Found a bottleneck - working with the database. How to deal with this?

Experience first: moki

Mock in PhpUnit is a dynamically created object whose class is dynamically inherited from the parodied class. You can configure what the methods of the moq will return, you can check which methods of the moq how many times with which parameters were called

The main plus of moki - they allow you to cut off whole pieces of functionality. Replacing the service with moch, we get rid of the need to think what is happening there, to develop additional scenarios and fixtures so that everything works correctly. As a result: fewer fixtures, and the response speed is higher due to the fact that we cut off the extra code that executes queries to the database.

The implicit plus of mobs is that they make them better organize dependencies. When you write a code, knowing that it will be necessary to write a test on it, where something is replaced by mokami, you immediately think about dependencies.

Minus : the test code is too attached to implementation. During the test, we must create a mock object and think about what methods should be called on it.

The second minus found is that the tests have become less reliable. They “do not notice” even changes in the interface, not to mention the implementation. Those. we deleted the method somewhere and after a long time found that the tests covering it still work as if nothing had happened, because we saw its mock, and he pretended that everything was fine.

I consider the experience with mokas unsuccessful in terms of speeding up the tests.

Experience Two: SQLite

The next option is SQLite DBMS , it can create a database in RAM. I had to write a PostgreSQL translator schema in SQLite, after each migration a new SQLite schema was generated. Tests from this circuit created an empty database in RAM. This approach increased the speed of tests on local machines by two to four times. It became realistically to run the entire test suite several times per hour.

But there were also disadvantages. We have lost many of the native PostgreSQL features (json, some handy aggregate functions, and more). Queries had to be written so that they worked on both PostgreSQL and SQLite.

Experience Three: PostgreSQL Optimization

This solution was working, but it caused some pain. At some point, we learned that PostgreSQL can be optimized for autotests, which reduces the response time by about four times. To do this, add a few settings to postgresql.conf:

fsync=off synchronous_commit=off full_page_writes=off 

These are reliability settings, they guarantee that if the server dies in the middle of a transaction, it will complete correctly when everything starts working again. It is clear that such settings cannot be made on production, but it was convenient on automatic tests.

This setting is applied to the entire cluster, affects all databases, it cannot be applied to any one database. If you manage to localize the databases in a separate cluster and disable fsync in it, this is very convenient.

A bit about new

I would also like to mention the danger of the new operator. Services created with its help cannot be replaced by mokas and stubs. Conclusion:

• Do not use new to create objects that are essentially services.
• It can be used in factories, because they can be replaced. But factories themselves should not be created through new .
• It can be used to create models, entities, DTO (data transfer object), value-objects.

Conclusions from three years of experience

• At the previous work, we refused acceptance tests, but now I would try them again: most likely many bugs were fixed in web drivers.
• If you need to cover new functionality with tests, you need to write only functional tests of controllers / components. In this situation, we have a high risk of structural changes, unit tests are unstable to them.
• There should not be many such tests, because many == slowly, they do not work as fast as modular ones. It is worth covering only those cases that can “shoot” (they have the likelihood of an error in the future).
• Unit tests are written on algorithm-rich methods (complex logic that needs to be tested) or on methods with little risk of structural changes in the future.
• Cons of mokas in general exceed pluses. It makes sense to use them only as a substitution of gateways into external APIs, and sometimes services from legacy code, which are very difficult to test.
• If you decide to write code without a test, it is advisable to think, “when you create it, what if in the future we still want to write a test for it?”
• Tests should be easy and pleasant to write, they give reliability, confidence, help to better understand the code, manage dependencies.
• Pay attention to the readability of the tests. It is necessary to relate to the test code in the same way as to the code that it covers.
• DB fixtures - part of the test, must also be readable