Airbnb testing

In a small startup, changing team behavior is relatively easy. You can sit down with the team at one table, discuss the new model of work, and then implement everything. And when you have dozens of engineers scattered across different projects, the transformation will require a more thorough strategy. One approach is to manage the process through direct orders “go and do it,” however, in the conditions of the relative independence of employees inherent in many startups, this approach does not work well. Another way is to personally set an example and indicate the direction of development.

From the first days in Airbnb, what happened with many startups happened: we wrote a lot of dubious code and spent little time on testing. And the more we grew, the more it became a problem. Today we process the data of millions of users daily. Just imagine how much data you have to deal with, given the payments alone, when transactions take place every minute in dozens of different currencies. A small minor error can have huge consequences for our users. Now, thanks to our tests, we can be sure that the final product will come out really high-quality.
')
We will tell you how we managed to change the testing culture and what tools we used to write and run tests to simplify the life of our team.



For us, a new testing culture began with pull requests (“change incorporation requests”). Once, several employees decided to tell others, using pull requests, about the changes that were made. This did not become a compulsory practice, but over time entailed a number of qualitative changes: first, the quality of the code increased, and second, it became simply old-fashioned.

This was facilitated by the growth of the team - each new engineer was informed about the importance of peer reviews (hereinafter PR), the percentage of PR in teams grew, regardless of whether the “old guard” switched to a new work model or not. As a result, even the “old men” began to feel uneasy, making changes directly to the master.

We started to take care of testing, actively promoted the development methodology through testing (Test-Driven development, TDD) and followed the three laws of TDD:

1. A new working code is written only after the “falling” unit test is written.
2. You write exactly such amount of unit test code as you need for this test to “fall” or not to compile.
3. You write exactly the amount of working code that is needed to pass the “falling” unit test.

A year later, significant progress was achieved. We started with a few weak tests, and came to a large set. Most importantly, we have gone from the vicious practice in which most of the new code gave up without any test at all. We not only became a team that believed in the tests, but became the ones who actually wrote them.

Using PR gives a number of advantages: we have a platform for discussing the structure of the code and architectural solutions, increased the likelihood that typos and logical errors will be found before they reach the users. PR allows all team members to better understand what we are working on and, in turn, leads to a further transformation of the entire testing culture.



Airbnb has a basic Rails test environment installed. A copy of the applications on the "rails" was stored within the directory / spec. Testing started rather slowly, and most of the employees did not know how to organize testing in their own local directory. New employees agreed that testing was essential, but when they saw that no one paid any attention to the tests, they quickly forgot about them. It was clear that it was necessary to make serious adjustments to the testing strategy.

First, we needed a lesson on how to deploy PR, including specific examples of work, be it testing a new function, correcting errors or refactoring. Secondly, we had to start educating the team, arrange meetings for engineers, show teammates how to write tests. The exchange of references and recommendations on the training literature were welcomed. So the company got a weekly newsletter with testing news, a demonstration of well-written specifications and an explanation of the highlights of PR.

Thirdly, we had to use our development as a tool for turning employees into real testing champions. This question is solved "in words", emphasizing the importance of testing on the example of the progress that has already been made. When testing volumes increased, other inertia employees joined the process and began to take the tests for granted.

Obviously, all these changes would not have been worth it if writing and executing tests had become too painful and difficult. In the next section, we will see how we managed to make the “cultural shift” less difficult for employees.



We needed to have a simple and fast testing environment, so we chose Vagrant . This utility allows you to create and configure a virtual development environment and supports multiple virtualization systems right out of the box. In conjunction with Chef - utilities for configuration management and much more - can be used to create VirtualHost and new project folders. We like the speed of this solution, and most of the people on our team use Zeus , which preloads the application to Rails to speed up its launch. Vagrant and Zeus have their own problems, but in practice we found that they save a tremendous amount of time.

There are many excellent solutions for accelerating Rails, but given the size of our code base and the speed with which we are evolving, most of them could not be applied immediately. We needed an assembly system that would allow us to parallelize our tests in real time. Our team used several different solutions before settling on Solano . Each of the previous systems revealed some problems: instability, memory problems, poor paralleling, a “scary” web interface, etc. Solano supports testing in multiple streams, running them in parallel, and then collecting the results. The product has impressive performance, an excellent web interface, there is a command line support. After the start of its use, the number of screaming and suffering gifs from frustrated engineers became record low.

The introduction of continuous integration (after each commit) was a huge first step, but we all wanted our testing to go perfectly. The reality of automated testing has shown that compromises are needed in testing cleanliness. This is normal. It is important that testing goes on all the time. Even when it is difficult. Especially when it's hard.

With the growth of Airbnb, we started adding much more critical features to the service. Obviously, there is still much room for improving automated testing, but now we are pleased with the result. We would never have been able to achieve this without the efforts shown “from below”, and these efforts would have been in vain if they had not been combined into a single strategy for improving the testing tools. All initiatives, in conjunction with the educational program, led us to a point where testing is ubiquitous and relatively painless for the system. The trick is that you need a team open to the idea of ​​developing good habits when testing.