We expand automation in a couple of hours: PHPUnit, Selenium, Composer

Hi, Habr! My name is Vitaliy Kotov, I work in Badoo, in the QA department. Most of the time I do test automation. Recently, I was faced with the task of deploying Selenium tests as quickly as possible for one of our projects. The condition was simple: the code should lie in a separate repository and not use the best practices of previous autotests. Oh yes, and it was necessary to do without CI. In this case, the tests should be run immediately after changing the project code. The report was supposed to come in the mail.

Actually, I decided to share the experience of such deployment. It turned out a kind of guide "How to run tests in a couple of hours."

Go!
')

Conditions of the problem

First of all, the task should be decomposed into several subtasks. It turns out that our mission, if we take up its execution, is as follows:

• need a separate repository;
• there must be tests;
• there must be a mechanism in it that will run tests to change the project code;
• The report should be readable, convenient and come to the mail to the specified people.

It seems everything is clear.

Stack

In Badoo, the first Selenium tests were written in PHP based on the PHPUnit framework. The Badoo server was mostly written in PHP, and by the time the automation appeared, it was decided not to produce technologies.

At that time, a framework from Facebook was chosen to work with Selenium, but at some point we were so enthusiastic about adding our functionality there, so that our version is no longer compatible with them.

Since the task was urgent, I decided not to experiment with technology. Unless the Facebook version of the latest version was chosen - it was interesting that there was something new there.

I downloaded the composer , with which it was more convenient for me to build such a project:

wget https://phar.phpunit.de/phpunit.phar 

The composer.json file looked like this then:

 { "require-dev": { "phpunit/phpunit": "5.3.*", "facebook/webdriver": "dev-master" } } 

Class MyTestCase

The first thing to do is to write your TestCase class:

 require_once __DIR__ . '/../../vendor/autoload.php'; class MyTestCase extends \PHPUnit_Framework_TestCase 

It introduced the setUp and tearDown functions, which created and killed the Selenium session, and the onNotSuccessfulTest function, which processed the data of the fallen test:

  /** @var RemoteWebDriver $driver */ protected $driver; protected function setUp() {} protected function tearDown() {} protected function onNotSuccessfulTest($e) {} 

Everything in setUp is quite simple: we create a session by specifying the URL of the Selenium farm and the desired capabilities. At this stage, I was only interested in the browser on which we were going to drive the tests.

  protected function setUp() { $this->driver = RemoteWebDriver::create( 'http://selenium-farm:5555/wd/hub', [WebDriverCapabilityType::BROWSER_NAME => WebDriverBrowserType::FIREFOX] ); } 

C tearDown is a bit trickier.

  protected function tearDown() { if ($this->driver) { $this->_prepareDataOnFailure(); $this->driver->quit(); } } 

The bottom line is this. For a dropped test, tearDown is performed before onNotSuccessfulTest is executed. Therefore, if we want to close the session in tearDown, all the necessary data from it should be obtained in advance: the current location, screenshot, and HTML snapshot, cookie values, and so on. All these data will be required by us for the formation of a beautiful and understandable report.

Collect data, respectively, should only be for dropped tests, remembering that tearDown will be called for all tests, including successfully passed, skipped and incomplete.

You can do it something like this:

  private function _prepareDataOnFailure() { $error_and_failure_statuses = [ PHPUnit_Runner_BaseTestRunner::STATUS_ERROR, PHPUnit_Runner_BaseTestRunner::STATUS_FAILURE ]; if (in_array($this->getStatus(), $error_and_failure_statuses)) { $this->data['url'] = $this->driver->getCurrentURL(); $ArtifactsHelper = new ArtifactsHelper($this->driver); $this->data['screenshot'] = $ArtifactsHelper->takeLocalScreenshot($this->current_test_name); $this->data['source'] = $ArtifactsHelper->takeLocalSource($this->current_test_name); } } 

The class ArtifactsHelper, as you might guess from the name, helps to collect artifacts. But about him a little later. :)

In the meantime, back to tearDown. In it, we have already collected all the necessary data for the fallen test, so that you can safely close the session.

This is followed by onNotSuccessfulTest, where we will need ReflectionClass . It looks like this:

 protected function onNotSuccessfulTest($e) { //prepare message $message = $this->_prepareCuteErrorMessage($e->getMessage()); //set message $class = new \ReflectionClass(get_class($e)); $property = $class->getProperty('message'); $property->setAccessible(true); $property->setValue($e, PHP_EOL . $message); parent::onNotSuccessfulTest($e); } 

In the exception message, we add all the information that we collected before closing the session. So it will be much more convenient for us to understand the reason for the drop in tests.

Class ArtifactsHelper

The class is pretty simple, so I’ll talk about it very briefly. It creates files that are called as a dropped test plus a timestamp, and puts them in the appropriate daddy. After running all the tests, the files are added to the final email with a report and deleted. This case looks like this:

 class ArtifactsHelper { const ARTIFACTS_FOLDER_PATH = __DIR__ . '/../artifacts/'; /** @var RemoteWebDriver */ private $driver; public function __construct(RemoteWebDriver $driver) { if (!is_dir(self::ARTIFACTS_FOLDER_PATH)) { mkdir(self::ARTIFACTS_FOLDER_PATH); } $this->driver = $driver; } public function takeLocalScreenshot($name) { if ($this->driver) { $name = self::_escapeFileName($name) . time() . '.png'; $path = self::ARTIFACTS_FOLDER_PATH . $name; $this->driver->takeScreenshot($path); return $path; } return ''; } public function takeLocalSource($name) { if ($this->driver) { $name = self::_escapeFileName($name) . time() . '.html'; $path = self::ARTIFACTS_FOLDER_PATH . $name; $html = $this->driver->getPageSource(); file_put_contents($path, $html); return $path; } return ''; } private static function _escapeFileName($file_name) { $file_name = str_replace( [' ', '#', '/', '\\', '.', ':', '?', '=', '"', "'", ":"], ['_', 'No', '_', '_', '_', '_', '_', '_', '', '', '_'], $file_name ); $file_name = mb_strtolower($file_name); return $file_name; } } 

In the constructor, we create the desired directory, if not, and bind the driver to the local field, so that it is more convenient to use it.

The takeLocalScreenshot and takeLocalSource methods create files with a screenshot (.png) and an HTML snapshot (.html). They will be called the test name, but we will replace some of the characters with others, so that the file name does not confuse the file system.

Tests

Tests will be inherited from MyTestCase. I will not give examples - everything is standard. Through $ this-> driver we work with Selenium, and all assertions and so on are executed through $ this.

It is worth saying a few words about passing parameters for running tests. PHPUnit will not add an unknown parameter to it when the test is run from the console. And it would be very convenient, for example, to be able to set the desired browser for tests.

I solved this problem in the following way: I created a bin / daddy in the project root, where I put an executable file called phpunit with the following content:

 #!/local/php/bin/php <?php require_once __DIR__ . '/../vendor/autoload.php'; require_once __DIR__ . '/../lib/MyCommand.php'; MyCommand::main(); 

And in the MyCommand class, respectively, I registered the desired parameters:

 class MyCommand extends PHPUnit_TextUI_Command { protected function handleArguments(array $argv) { $this->longOptions['platform='] = null; $this->longOptions['browser='] = null; $this->longOptions['local'] = null; $this->longOptions['proxy='] = null; $this->longOptions['send-report'] = null; parent::handleArguments($argv); } } 

Now, if we run tests from our phpunit file, we can set parameters that will be passed to tests in the $ GLOBALS ['argv'] array. Then you can parse it and somehow handle it.

Run to change the project code

So, now we have everything to start running tests on the trigger. Unfortunately, we do not have a repository with the project code, so it is not possible to find out when the changes were made in it. Without the help of developers here is not enough.

We agreed that in the test environment the application will have a special address where you can see the last commit hash (in fact, the version of the site).

Then everything is simple: by cron, we launch a special script script every couple of minutes. When you first start it goes with Curl to this address and gets the current version of the site. Then he creates in the special directory the file version.file, where he writes this version. The next time he gets the version from the site and from the file; if they are different, write the new version to the file and run the tests. If not, it does nothing.

In the end, it looks like this:

 function isVersionChanged($domain) { $url = $domain . 'version'; $ch = curl_init(); curl_setopt($ch, CURLOPT_URL, $url); curl_setopt($ch, CURLOPT_RETURNTRANSFER, 1); if ($proxy = SeleniumConfig::getInstance()->getProxy()) { curl_setopt($ch, CURLOPT_PROXY, $proxy); } $response = curl_exec($ch); curl_close($ch); $version_from_site = trim($response); $version_from_file = file_get_contents(VERSION_FILE); return ($version_from_site != $version_from_file); } 

Sending a letter with a report

Unfortunately, in PHPUnit from the TestCase class it is impossible to determine whether the last test was in the suite or not. Of course, there is a method tearDownAfterClass, but it is executed after the completion of tests in one class. If, for example, two classes with tests are specified in the suite, tearDownAfterClass will be executed twice.

I also needed to register the logic somewhere, which will send a letter guaranteed after passing all the tests. And, of course, do it only once. As you may have guessed, I wrote another helper. :)

Class mailer

This class stores information about past tests: error texts, paths to a file with a screenshot and an HTML snapshot. It is made on the principle of Singleton, instantiated once at the first call. And not forcibly destroyed. Do you understand what I am doing? :)

  public function __destruct() { if ($this->send_email) { $this->sendReport($this->tests_failed, $this->tests_count); } } private function sendReport(array $report, $tests_count) { $count = count($report); $is_success_run = $count == 0; // start message if ($is_success_run) { $message = "All tests run successfully! Total amount: {$tests_count}."; $subject = self::REPORT_SUBJECT_SUCCESS; } else { $message = "Autotests failed for project. Failed amount: {$count}, total amount: {$tests_count}."; $subject = self::REPORT_SUBJECT_FAILURE; } $message .= PHP_EOL; $start_version = VersionStorage::getInstance()->getStartVersion(); $finish_version = VersionStorage::getInstance()->getFinishVersion(); if ($start_version == $finish_version) { $message .= 'Application version: ' . $start_version . PHP_EOL; foreach ($report as $testname => $text) { $message .= PHP_EOL . $testname . PHP_EOL . trim($text) . PHP_EOL; } } else { $message .= PHP_EOL; $message .= "***APPLICATION VERSION HAS BEEN CHANGED***" . PHP_EOL; $message .= "Version on start: {$start_version}" . PHP_EOL; $message .= "Current version: {$finish_version}" . PHP_EOL; $message .= "TESTS WILL BE RE-LAUNCHED IN FEW MINUTES."; $subject = self::REPORT_SUBJECT_FAILURE; } // end message foreach (self::$report_recipients as $email_to) { $this->_sendMail($email_to, self::EMAIL_FROM, $subject, $message); } 

That's right, the logic of sending a letter I added to the destructor. When the test run process completes, the garbage collector arrives and destroys my class. Therefore, the destructor is triggered at the very last moment.

Depending on whether the tests passed successfully or not, the title of the letter changes. If the site version has changed during the run, the tests are run again.

Autopool Test

And finally, a bit of convenience. Since the whole system will live somewhere on a remote server, it would be convenient if she could do git pull herself, so as not to forget to freeze important changes in the tests.

To do this, create an executable file with the following content:

 #! /usr/bin/env bash cd `dirname "$0"` output=$(git -c gc.auto=0 pull -q origin master 2>&1) if [ ! $? -eq 0 ]; then echo "${output}" | mail -s "Failed to update selenium repo on selenium-server" username@corp.badoo.com fi 

The script will execute the git pull command, and if something goes wrong and it fails, it will write a letter to the responsible employee.

Then we add the script to cron, running every couple of minutes, and the trick is in the hat.

Results

The results are usually summed up with an eye on the original task. Here is what we get:

• a separate repository has appeared;
• there with the help of composer we compiled a project: downloaded PHPUnit and the Facebook framework;
• wrote their TestCase-class, which is able to generate convenient reports;
• wrote tests that can be run in different browsers and with different parameters;
• created a mechanism that will run these tests when changing the version of the test project;
• took care of sending a letter with a report and screenshots;
• added a script that automatically updates this whole thing to the required version.

It seems nothing is missed.

Such is the story. Thanks for attention! I will be glad to hear your stories, write in the comments. :)