We cover the project with smoke tests until it burns out

Hi, Habr! Once, at our internal seminar, my manager, the head of the testing department, began his speech with the words “testing is not necessary.” In the hall, everyone was quiet, some even tried to fall from the chairs. He continued his thought: without testing it is quite possible to create a complex and expensive project. And, most likely, it will work. But imagine how much more confident you will feel, knowing that the product works as it should.

In Badoo releases occur quite often. For example, the server part, along with the desktop web, is released twice a day. So we know firsthand that complex and slow testing is a stumbling block to development. Fast testing is happiness. So, today I will talk about how smoke-testing is arranged at Badoo.

What is smoke testing

The first use of this term was in the stove-makers, who, having assembled the stove, closed all the plugs, flooded it and watched the smoke go only from the proper places. Wikipedia
')
In its original application, smoke testing is designed to test the simplest and most obvious cases, without which any other type of testing would be unnecessarily unnecessary.

Let's look at a simple example. The pre-production of our application is located at bryak.com (any coincidences with real sites are random). We have prepared and uploaded a new release for testing. What is worth checking out first? I would start by checking that the application is still open. If the web server responds to “200”, then everything is fine and you can begin to check the functionality.

How to automate such a check? In principle, you can write a functional test that will raise the browser, open the desired page and make sure that it is displayed as it should. However, this solution has a number of minuses. First, it takes a long time: the browser launch process will take longer than the check itself. Secondly, this requires the maintenance of additional infrastructure: for the sake of such a simple test, we will need to hold a server with browsers somewhere. Conclusion: it is necessary to solve the problem differently.

Our first smoke test

In Badoo, the server part is written mostly in PHP. Unit tests for obvious reasons are written on it. Total we already have PHPUnit. In order not to produce technologies unnecessarily, we decided to write smoke tests in PHP too. In addition to PHPUnit, we need a client library to work with the URL (libcurl) and the PHP extension to work with it - cURL.

In fact, the tests simply make the necessary requests to the server and check the answers. Everything is tied up with the getCurlResponse () method and several types of asserts.

The method itself looks like this:

public function getCurlResponse( $url, array $params = [ 'cookies' => [], 'post_data' => [], 'headers' => [], 'user_agent' => [], 'proxy' => [], ], $follow_location = true, $expected_response = '200 OK' ) { $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_HEADER, 1); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); if (isset($params['cookies']) && $params['cookies']) { $cookie_line = $this->prepareCookiesDataByArray($params['cookies']); curl_setopt($ch, CURLOPT_COOKIE, $cookie_line); } if (isset($params['headers']) && $params['headers']) { curl_setopt($ch, CURLOPT_HTTPHEADER, $params['headers']); } if (isset($params['post_data']) && $params['post_data']) { $post_line = $this->preparePostDataByArray($params['post_data']); curl_setopt($ch, CURLOPT_POST, 1); curl_setopt($ch, CURLOPT_POSTFIELDS, $post_line); } if ($follow_location) { curl_setopt($ch, CURLOPT_FOLLOWLOCATION, 1); } if (isset($params['proxy']) && $params['proxy']) { curl_setopt($ch, CURLOPT_PROXY, $params['proxy']); } if (isset($params['user_agent']) && $params['user_agent']) { $user_agent = $params['user_agent']; } else { $user_agent = USER_AGENT_DEFAULT; } curl_setopt($ch, CURLOPT_USERAGENT, $user_agent); curl_setopt($ch, CURLOPT_AUTOREFERER, 1); $response = curl_exec($ch); $this->logActionToDB($url, $user_agent, $params); if ($follow_location) { $this->assertTrue( (bool)$response, 'Empty response was received. Curl error: ' . curl_error($ch) . ', errno: ' . curl_errno($ch) ); $this->assertServerResponseCode($response, $expected_response); } curl_close($ch); return $response; } 

The method itself is able to return the server response at a given URL. At the input accepts parameters such as cookies, headers, user agent and other data necessary for the formation of the request. When a response from the server is received, the method checks that the response code is the expected one. If this is not the case, the test fails with an error reporting this. This is done to make it easier to determine the cause of the fall. If the test falls on an assertion, telling us that there is no element on the page, the error will be less informative than the message that the response code, for example, “404” instead of the expected “200”.

When the request is sent and the response is received, we log the request so that, if necessary, in the future, it is easy to reproduce the chain of events if the test fails or breaks. I will tell about it below.

The simplest test looks like this:

 public function testStartPage() { $url = 'bryak.com'; $response = $this->getCurlResponse($url); $this->assertHTMLPresent('<body>', $response, 'Error: test cannot find body element on the page.'); } 

This test passes in less than a second. During this time, we checked that the start page responds with “200”, and there is a body element on it. With the same success, we can check any number of elements on the page, the test duration will not change significantly.

Advantages of such tests:

• speed - the test can be run as often as necessary. For example, for every code change;
• do not require special software and hardware for work;
• they are easy to write and maintain;
• they are stable.

Concerning the last point. I mean - no less stable than the project itself.

Authorization

Imagine that it has been three days since we wrote our first smoke test. Of course, during this time we have covered all unauthorized pages, which we have just found, with tests. We sat for a while and were glad, but then we realized that all the most important things in our project are beyond authorization. How to get the opportunity to test it too?



What is the difference between an authorized page and an unauthorized page? From the point of view of the server, everything is simple: if the request contains information on which the user can be identified, an authorized page will be returned to us.

The easiest option is an authorization cookie. If you add it to the request, the server will “recognize” us. Such a cookie can be hard-coded in a test if its lifetime is rather long, and you can receive it automatically by sending requests to the authorization page. Let's take a closer look at the second option.

On our website, the login page looks like this:



We are interested in the form where you need to enter the username and password of the user.

Open this page in any browser and open the inspector. Enter user data and submit the form.

In the inspector there was a request that we need to imitate in the test. You can see what data, in addition to the obvious (login and password), are sent to the server. For each project in different ways: it can be remote token, data from any cookies received earlier, user agent, and so on. Each of these parameters will have to first get in the test before you create a request for authorization.

In the developer’s tools of any browser, you can copy the request by selecting the copy as cURL option. In this form, the command can be inserted into the console and viewed there. It can also be tested there by changing or adding parameters.



In response to such a request, the server will return us cookies, which we will add to further requests in order to test authorized pages.

Since authorization is a rather long process, I suggest to get an authorization cookie only once for each user and save somewhere. We have, for example, such cookies are stored in an array. The key is the username, and the value is information about them. If there is no key for the next user, log in. If there is, we make the request we are interested in immediately.

An example of a test code checking an authorized page looks like this:

 public function testAuthPage() { $url = 'bryak.com'; $cookies = $this->getAuthCookies('employee@bryak.com', '12345'); $response = $this->getCurlResponse($url, ['cookies' => $cookies]); $this->assertHTMLPresent('<body>', $response, 'Error: test cannot find body element on the page.'); } 

As we can see, a method has been added that receives an authorization cookie and simply adds it to a further request. The method itself is implemented quite simply:

 public function getAuthCookies($email, $password) { // check if cookie already has been got If (array_key_exist($email, self::$known_cookies)) { return self::$known_cookies[$email]; } $url = self::DOMAIN_STAGING . '/auth_page_adds'; $post_data = ['email' => $email, 'password' => $password]; $response = $this->getCurlResponse($url, ['post_data' => $post_data]); $cookies = $this->parseCookiesFromResponse($response); // save cookie for further use self::$known_cookies[$email] = $cookies; return $cookies; } 

The method first checks if there is a login cookie already received for this e-mail (in your case it may be a login or something else). If it is, it returns it. If not, he makes a request for an authorization page (for example, bryak.com/auth_page_adds) with the necessary parameters: e-mail and user password. In response to this request, the server sends headers, among which are the cookies we are interested in. It looks like this:

 HTTP/1.1 200 OK Server: nginx Content-Type: text/html; charset=utf-8 Transfer-Encoding: chunked Connection: keep-alive Set-Cookie: name=value; expires=Wed, 30-Nov-2016 10:06:24 GMT; Max-Age=-86400; path=/; domain=bryak.com 

Using these simple headers, we need to get the name of the cookie and its value (in our example, this is name = value). We have a method that parses the answer, looks like this:

 $this->assertTrue( (bool)preg_match_all('/Set-Cookie: (([^=]+)=([^;]+);.*)\n/', $response, $mch1), 'Cannot get "cookies" from server response. Response: ' . $response ); 

After cookies are received, we can safely add them to any request to make it authorized.

Parsing Falling Tests

From the above, it follows that such a test is a set of server requests. We make a request, we make a manipulation with the answer, we make the next request and so on. The thought creeps into his head: if such a test falls on the tenth request, it may be difficult to understand the reason for its fall. How to simplify your life?

First of all, I would like to advise you to maximally atomize the tests. It is not necessary to check 50 different cases in one test. The simpler the test, the easier it will be with it in the future.

Still useful to collect artifacts. When our test fails, it saves the last server response in an HTML file and uploads it to the artifact storage, where this file can be opened from the browser, specifying the name of the test.

For example, our test fell on the fact that it could not find a piece of HTML on the page:

 <span class=”link”>Link<span> 

We go to our collector and open the corresponding page:



You can work with this page in the same way as with any other HTML page in the browser. You can use the CSS locator to try to find the missing element and, if it really does not exist, decide that it has either changed or been lost. Maybe we found a bug! If the element is in place, it is possible that we somewhere were mistaken in the test - we must carefully look in this direction.

Logging helps to simplify life. We try to log all requests that the dropped test did, so that they can be easily repeated. Firstly, it allows you to quickly make a set of similar actions with your hands to reproduce an error, secondly, to identify frequently falling tests, if we have such.

In addition to helping with error analysis, the logs described above help us generate a list of authorized and unauthorized pages that we tested. Looking at it is easy to find and eliminate spaces.

Last but not least, I can advise - the tests should be as convenient as possible. The easier it is to run them, the more often they will be used. The clearer and more concise the report about the fall, the more closely it will be studied. The simpler the architecture, the more tests will be written and the less time it will take to write a new one.

If it seems to you that it is inconvenient to use tests - most likely you do not think so. It is necessary to fight this as soon as possible. Otherwise, you run the risk of starting to pay less attention to these tests at some point, and this may already lead to an error in production.



In words, the idea seems obvious, I agree. But in fact, we all have to strive for. So simplify and optimize your creations and live without bugs. :)

Results

At the moment, we have * opening Timitsiti * th, already 605 tests. All tests, if not run in parallel, pass in a little less than four minutes.

During this time, we are convinced that:

• our project opens in all languages ​​(of which we have more than 40 in production);
• For major countries, the correct payment forms are displayed with the appropriate set of payment methods;
• basic API requests work correctly;
• Landing page for redirects works correctly (including on a mobile site with a corresponding user agent);
• all internal projects are displayed correctly.

Tests on Selenium WebDriver for all of this would take many times more time and resources.

Of course, this is not a replacement for Selenium. We still have to check the correct behavior of the client and cross-browser cases. We can replace only those tests that check the server behavior. But beyond that, we can carry out preliminary testing, quick and easy. If at the stage of smoke testing there were errors and “the smoke is not coming from there,” perhaps it is not worth running a long set of heavy Selenium tests until fixes? This is up to you! :)

Thanks for attention.

Vitaly Kotov, QA-automation engineer.