Selenium Manager: the story of a single interface
Hi, Habr!
My name is Vitaliy Kotov and I work for Badoo. In a previous article, I told you that we have a certain interface that helps testers and developers interact with autotests.
Not once or twice I was asked to tell about it in more detail.
')
Under the cut, I (finally!) Will talk about how this interface was written and what it can do. I'll tell you about the features that have taken root, and about those that were unclaimed for one reason or another. Perhaps some ideas will seem interesting to you, and you will also think about such an “assistant”.
So, it all started with the Selenium tests. Hence the name of the interface - Selenium Manager. Although now it can also run Smoke tests .
At some point, we realized that our tests can do a lot of things. And it became clear by the number of additional keys that could be specified at launch, so that the tests were performed in one way or another. Here are just some of them:
And so on. In total, I counted 25 keys. And for 11 of them, you can specify more than two options. This is a lot, and I did not want to keep it all in my head or use the “Help” command without end.
The first version of the interface appeared in 2014. According to my idea, it should just parse all possible keys and values ​​from the config and draw the corresponding set of HTML elements in the browser. After the user set the required set of parameters and clicked on the “get start command” button, he saw on the screen the corresponding line, which could be copied to the terminal.
During the evening, “throwing around” the prototype, I went to show it to the guys. Of course, everyone liked it, “but maybe you can make the test run immediately from this interface?” Was the answer. Quite expected, of course.
I started thinking this way ...
Our tests are written in PHP. As a start-up, we use PHPUnit. The idea to run the test from the console was as follows: we set the required set of parameters, specify the test and send an AJAX request to the server where our test code will be executed.
On the server side, a command string is formed that will be run using the exec command. Do not forget to untie the command from the console, otherwise the exec will be executed for as long as the test itself is executed. We do not need this - we only need to get the PID process.
Like that:
As can be seen from the context, we first create a logfile, where we will write a log in the course of passing the test.
After receiving the PID, we return it and the path to the log file to the client, which runs a simple setInterval . Every N seconds, the client knocks on the server and receives the actual contents of the log file and the PID status. If the PID is gone, clearInterval is called . Quite simply:
Thus, we see the progress of the test. And we know when it ended.
Back in 2013, PHPUnit could not run tests in parallel. And this was a serious problem, because even then we had so many Selenium tests that they could take more than one hour to flow. Then my colleague Ilya Kudinov wrote an article about how we began to solve this problem for unit tests.
But the final solution was not very suitable for Selenium tests.
The optimal solution was Selenium Manager. After all, if you can run one test from it, why not open a new tab and run the second one? Just kidding ...
This can be done from the same tab. To do this, it is enough to store on the client side a list of PIDs for each of the tests and, when requested by the server, poll them all, returning the status for each. With log files do the same.
I added to the interface the ability to set the number of tests run simultaneously. In fact, this is the number of threads in which tests will run. If the number of threads is less than the total number of tests, then the tests are queued and wait for one of the running tests to take its place.
This was not enough. Now it was necessary to open the terminal and copy the path from there to all the necessary files with tests. Moreover, sometimes you want to run not all the tests from the file, but only some (for this, there is a filter parameter in PHPUnit).
To solve this problem, I on the client made a simple HTML element textarea, where you could write only the names of the classes, and specify a filter using "::". For example:
There you could write anything and even copy a list of dropped tests from TeamCity. All this was sent to the server and parsed. How to determine if some combination is similar to the name of the class in which there is such a test? And then determine the path to the file with this class to run the test?
My method was like this:
We divide the content by space or transfer. Next, we process each of the parts in turn. In it, we first look for the "::" combination to get the value for the filter.
Next we look at the first part (up to the “::” sign). If it is specified in the form of the path to the file, run it. If not (for example, only the class was specified) - run the method findPathByClassName, which can search by class name in test folders and return the path to the required.
In PHPUnit it is possible to set the test group (about this here ). And, accordingly, to run tests, specifying these groups. We have these groups tied to the features on the site that these tests cover. Most often, when you need to run a lot of tests (but not all), you need to run them for some group or groups.
I added a list of groups, which is obtained interactively using the same search, with the ability to select several of them and run tests without specifying any paths and class names.
The interface for running tests in groups looks like this:
Now you can run tests either by groups or by the list of names of these tests.
The tests themselves run like this:
The test is marked yellow if it is already running; green - if it was successful, red - if it fell. Blue marked tests that were successful, but inside which there are skipped tests.
Inside the cells is the usual log that we see when running the test through PHPUnit:
Determining whether the test passed successfully or not can be quite simple on the client side using regular expressions:
This is not the most fail-safe solution. If we once stop using PHPUnit, or if the output string changes significantly, the method will stop working (we will definitely find out by getting the answer "-1"). But neither one nor the other in the near future, most likely, will not happen.
While I was doing the interface for running tests, we actively began to run tests for shots. It turned out that it was not easy to run as many tests as we wanted. There are not enough agents , but it is expensive.
Then we transferred the test run to the “cloud” system. About her, we may write a detailed article next time. In the meantime, I’ll dwell on the fact that, since this system is cloudy, it is problematic to collect logs from it.
We created a simple MySQL table, where the tests at the end of the run started to log the results: passed the test successfully or fell, with what error, how long it went, and so on.
Based on this table on another tab of Selenium Manager, we started to draw the results of the runs under the name of the shot. With the ability to restart the dropped test directly from the interface (if the test passes successfully, it disappears from the list of fallen) or find out if this test fell on other shots with the same error (determined by the error tracing):
You can group tests either by type of error (the same trace), or arrange in alphabetical order.
In addition to the pages described above, I would like to elaborate on another one. This is an investigate page. It so happens that an important task gets into testing with a minor bug. Let's say a JS error that does not affect the user. To roll back such a task is inappropriate. We proceed in the following way: we put a bug-tick on the developer, and we skip the task further.
But what to do with the dough, which will now honestly fall on staging and shots, making it difficult to live and littering the logs? We know why it falls, we know that until the task is done, it will continue to fall. Is it worth the time to run it and get the expected result?
I decided that the problem could be solved in the following way: all in the same MySQL to create a label to which a test with the indication of the ticket, due to which it is broken, will be added through the interface.
Tests before launch will receive this list and compare their name with the names from it. If the test does not need to be run, it will be marked as skipped with the corresponding task in the log.
I also wrote a simple script that goes to the crown on this table, gets a list of tasks and goes to JIRA (our bugtracker) for statuses for them. If a task is closed, the entry from the investigate table is deleted, the test starts automatically.
In addition to the above, Selenium Manager is able to still a bunch of interesting and equally useful things.
For example, get a list of all shots for the current build and run the specified test for each of them. This is extremely useful when the test began to fall on the pricing, and it was not possible to determine the guilty ticket by hand and eyes.
There is also a page with a table, where a list of unstable tests and the most frequent errors (for different tests). It is useful for automation engineers as it helps keep tests in order. Do not forget that Selenium tests, like any other UI tests, are unstable by definition, and this is normal. But to have the statistics of the most "bad" tests in order to stabilize them is good.
Opposite each test in this tablet there is a button with which you can create and transfer a test stabilization ticket to yourself in one click. The link to the ticket will remain in the table, so it will be seen if someone is already engaged in the test.
It looks like this:
There is a page on which live graphics nodes Selenium farm . Our automation engineer, Artyom Soldatkin, has already talked about how to patch Selenium, so that using an HTTP request it would be possible to obtain data on the number of free browsers grouped in these browsers and versions. More on this here .
I wrote a simple script that goes to the Selenium farm according to the crown and collects this information, putting it into a MySQL table. On the client side, I used plotly.js to draw graphs from this data.
It looks like this:
So there is an opportunity to find out whether we have enough capacity for all our needs. :)
Most often, it is impossible to say in advance whether the feature will be useful or not until you try. Sometimes a seemingly useful idea is unclaimed.
For example, we had a page where everyone could subscribe to a specific test or group of tests. If these tests started dropping on pricing, all those who subscribed received notifications to the chat.
Initially, it was thought that this would be useful for tasks that cannot be fully tested on a device or a shot, when a manual tester monitors the state of tests. By subscribing, he could understand that the test group suitable for the description broke down on the staging and that there were some problems with the task.
In fact, it turned out that it was difficult to predict a group of tests. Most of the time, there were a lot of unnecessary notifications, and sometimes the necessary notifications did not come because the fallen tests were from other groups.
As a result, we returned to the previous scheme, where automators themselves notify the guys from the testing department that there are any problems.
The interface has been successfully integrated into the testing process. Now you can easily, quickly and easily run tests with any parameters, monitor their stability and the entire system of autotests in general.
In general, I am pleased that I managed to get away from the console. Not because there is something bad in it, but simply because the interface in this case saves a lot of time that can be spent with benefit.
What conclusion can be made from this? Optimize and simplify the work with your tools - and you will be happy.
Thanks for attention.
My name is Vitaliy Kotov and I work for Badoo. In a previous article, I told you that we have a certain interface that helps testers and developers interact with autotests.
Not once or twice I was asked to tell about it in more detail.
')
Under the cut, I (finally!) Will talk about how this interface was written and what it can do. I'll tell you about the features that have taken root, and about those that were unclaimed for one reason or another. Perhaps some ideas will seem interesting to you, and you will also think about such an “assistant”.
How it all began
So, it all started with the Selenium tests. Hence the name of the interface - Selenium Manager. Although now it can also run Smoke tests .
At some point, we realized that our tests can do a lot of things. And it became clear by the number of additional keys that could be specified at launch, so that the tests were performed in one way or another. Here are just some of them:
- Platform - determines on which platform (dev, shot or staging) we run tests. About what stages of testing we have in the company can be read in this article .
- App - determines which of our applications will run a test. Since the test stores only business logic, separate from PageObject , this key is quite useful.
- Browser - sets the browser on which the test will run.
- Local - if you specify this key, the test will go to a locally raised Selenium server instead of a Selenium farm.
- Help - if you specify this key, instead of running the test, we will see instructions for all keys. :)
And so on. In total, I counted 25 keys. And for 11 of them, you can specify more than two options. This is a lot, and I did not want to keep it all in my head or use the “Help” command without end.
First version
The first version of the interface appeared in 2014. According to my idea, it should just parse all possible keys and values ​​from the config and draw the corresponding set of HTML elements in the browser. After the user set the required set of parameters and clicked on the “get start command” button, he saw on the screen the corresponding line, which could be copied to the terminal.
During the evening, “throwing around” the prototype, I went to show it to the guys. Of course, everyone liked it, “but maybe you can make the test run immediately from this interface?” Was the answer. Quite expected, of course.
I started thinking this way ...
Running a test from the interface
Our tests are written in PHP. As a start-up, we use PHPUnit. The idea to run the test from the console was as follows: we set the required set of parameters, specify the test and send an AJAX request to the server where our test code will be executed.
On the server side, a command string is formed that will be run using the exec command. Do not forget to untie the command from the console, otherwise the exec will be executed for as long as the test itself is executed. We do not need this - we only need to get the PID process.
Like that:
function launchTest($cmd, $logfile) { $cmd = escapeshellcmd($cmd); exec($command = 'sh -c "' . $cmd . '" > ' . $logfile . ' & echo $!', $out); if (count($out) !== 1) { die("Wrong launch command: {$command}"]); } return $out[0]; } As can be seen from the context, we first create a logfile, where we will write a log in the course of passing the test.
After receiving the PID, we return it and the path to the log file to the client, which runs a simple setInterval . Every N seconds, the client knocks on the server and receives the actual contents of the log file and the PID status. If the PID is gone, clearInterval is called . Quite simply:
var interval_id; function startPidProcessing() { interval_id = setInterval(self.checkPid, 4500); } function stopPidProcessing() { clearInterval(interval_id); interval_id = null; } Thus, we see the progress of the test. And we know when it ended.
Running multiple tests in parallel
Back in 2013, PHPUnit could not run tests in parallel. And this was a serious problem, because even then we had so many Selenium tests that they could take more than one hour to flow. Then my colleague Ilya Kudinov wrote an article about how we began to solve this problem for unit tests.
But the final solution was not very suitable for Selenium tests.
The optimal solution was Selenium Manager. After all, if you can run one test from it, why not open a new tab and run the second one? Just kidding ...
This can be done from the same tab. To do this, it is enough to store on the client side a list of PIDs for each of the tests and, when requested by the server, poll them all, returning the status for each. With log files do the same.
I added to the interface the ability to set the number of tests run simultaneously. In fact, this is the number of threads in which tests will run. If the number of threads is less than the total number of tests, then the tests are queued and wait for one of the running tests to take its place.
Running tests on filter and group
This was not enough. Now it was necessary to open the terminal and copy the path from there to all the necessary files with tests. Moreover, sometimes you want to run not all the tests from the file, but only some (for this, there is a filter parameter in PHPUnit).
To solve this problem, I on the client made a simple HTML element textarea, where you could write only the names of the classes, and specify a filter using "::". For example:
There you could write anything and even copy a list of dropped tests from TeamCity. All this was sent to the server and parsed. How to determine if some combination is similar to the name of the class in which there is such a test? And then determine the path to the file with this class to run the test?
My method was like this:
public static function parseTestNames($textarea) { $parts = preg_split('/[,\s\n]+/', $textarea); $tests = []; foreach ($parts as $part) { $part_splited = explode('::', $part); $filter = $part_splited[1] ?? false; $testname = $part_splited[0]; if (is_file($testname)) { $testname = $filter ? $testname . '::' . $filter : $testname; $tests[] = $testname; } else { $found_tests = self::findPathByClassName($testname, $filter, $hard); foreach ($found_tests as $test) { $test = $filter ? $test . '::' . $filter : $test; $tests[] = $test; } } } $result = array_values(array_unique($tests)); return $result; We divide the content by space or transfer. Next, we process each of the parts in turn. In it, we first look for the "::" combination to get the value for the filter.
Next we look at the first part (up to the “::” sign). If it is specified in the form of the path to the file, run it. If not (for example, only the class was specified) - run the method findPathByClassName, which can search by class name in test folders and return the path to the required.
In PHPUnit it is possible to set the test group (about this here ). And, accordingly, to run tests, specifying these groups. We have these groups tied to the features on the site that these tests cover. Most often, when you need to run a lot of tests (but not all), you need to run them for some group or groups.
I added a list of groups, which is obtained interactively using the same search, with the ability to select several of them and run tests without specifying any paths and class names.
The interface for running tests in groups looks like this:
Now you can run tests either by groups or by the list of names of these tests.
The tests themselves run like this:
The test is marked yellow if it is already running; green - if it was successful, red - if it fell. Blue marked tests that were successful, but inside which there are skipped tests.
Inside the cells is the usual log that we see when running the test through PHPUnit:
Determining whether the test passed successfully or not can be quite simple on the client side using regular expressions:
function checkTestReport(reg_exp) { let text = $(cell_id).html(); let text_arr = text.split("\n"); if (text_arr[0].search('PHPUnit') != -1) { return reg_exp.test(text_arr[2]); } else { return -1; } } this.isTestSuccessful = function(cell_id) { return checkTestReport(/^[\.]+\s/); }; this.isTestSkipped = function(cell_id) { return checkTestReport(/^[\.SI]+\s/); }; This is not the most fail-safe solution. If we once stop using PHPUnit, or if the output string changes significantly, the method will stop working (we will definitely find out by getting the answer "-1"). But neither one nor the other in the near future, most likely, will not happen.
Running tests in the "cloud"
While I was doing the interface for running tests, we actively began to run tests for shots. It turned out that it was not easy to run as many tests as we wanted. There are not enough agents , but it is expensive.
Then we transferred the test run to the “cloud” system. About her, we may write a detailed article next time. In the meantime, I’ll dwell on the fact that, since this system is cloudy, it is problematic to collect logs from it.
We created a simple MySQL table, where the tests at the end of the run started to log the results: passed the test successfully or fell, with what error, how long it went, and so on.
Based on this table on another tab of Selenium Manager, we started to draw the results of the runs under the name of the shot. With the ability to restart the dropped test directly from the interface (if the test passes successfully, it disappears from the list of fallen) or find out if this test fell on other shots with the same error (determined by the error tracing):
You can group tests either by type of error (the same trace), or arrange in alphabetical order.
Investigate system
In addition to the pages described above, I would like to elaborate on another one. This is an investigate page. It so happens that an important task gets into testing with a minor bug. Let's say a JS error that does not affect the user. To roll back such a task is inappropriate. We proceed in the following way: we put a bug-tick on the developer, and we skip the task further.
But what to do with the dough, which will now honestly fall on staging and shots, making it difficult to live and littering the logs? We know why it falls, we know that until the task is done, it will continue to fall. Is it worth the time to run it and get the expected result?
I decided that the problem could be solved in the following way: all in the same MySQL to create a label to which a test with the indication of the ticket, due to which it is broken, will be added through the interface.
Tests before launch will receive this list and compare their name with the names from it. If the test does not need to be run, it will be marked as skipped with the corresponding task in the log.
I also wrote a simple script that goes to the crown on this table, gets a list of tasks and goes to JIRA (our bugtracker) for statuses for them. If a task is closed, the entry from the investigate table is deleted, the test starts automatically.
What else can Selenium Manager
In addition to the above, Selenium Manager is able to still a bunch of interesting and equally useful things.
For example, get a list of all shots for the current build and run the specified test for each of them. This is extremely useful when the test began to fall on the pricing, and it was not possible to determine the guilty ticket by hand and eyes.
There is also a page with a table, where a list of unstable tests and the most frequent errors (for different tests). It is useful for automation engineers as it helps keep tests in order. Do not forget that Selenium tests, like any other UI tests, are unstable by definition, and this is normal. But to have the statistics of the most "bad" tests in order to stabilize them is good.
Opposite each test in this tablet there is a button with which you can create and transfer a test stabilization ticket to yourself in one click. The link to the ticket will remain in the table, so it will be seen if someone is already engaged in the test.
It looks like this:
There is a page on which live graphics nodes Selenium farm . Our automation engineer, Artyom Soldatkin, has already talked about how to patch Selenium, so that using an HTTP request it would be possible to obtain data on the number of free browsers grouped in these browsers and versions. More on this here .
I wrote a simple script that goes to the Selenium farm according to the crown and collects this information, putting it into a MySQL table. On the client side, I used plotly.js to draw graphs from this data.
It looks like this:
So there is an opportunity to find out whether we have enough capacity for all our needs. :)
What features did not take root
Most often, it is impossible to say in advance whether the feature will be useful or not until you try. Sometimes a seemingly useful idea is unclaimed.
For example, we had a page where everyone could subscribe to a specific test or group of tests. If these tests started dropping on pricing, all those who subscribed received notifications to the chat.
Initially, it was thought that this would be useful for tasks that cannot be fully tested on a device or a shot, when a manual tester monitors the state of tests. By subscribing, he could understand that the test group suitable for the description broke down on the staging and that there were some problems with the task.
In fact, it turned out that it was difficult to predict a group of tests. Most of the time, there were a lot of unnecessary notifications, and sometimes the necessary notifications did not come because the fallen tests were from other groups.
As a result, we returned to the previous scheme, where automators themselves notify the guys from the testing department that there are any problems.
Results
The interface has been successfully integrated into the testing process. Now you can easily, quickly and easily run tests with any parameters, monitor their stability and the entire system of autotests in general.
In general, I am pleased that I managed to get away from the console. Not because there is something bad in it, but simply because the interface in this case saves a lot of time that can be spent with benefit.
What conclusion can be made from this? Optimize and simplify the work with your tools - and you will be happy.
Thanks for attention.
Source: https://habr.com/ru/post/344030/
All Articles