How we test CSS regressions with Gemini. Report on BEMup in Yandex
Hello! My name is Sergey Tatarintsev. In Yandex, I work in a group of common interfaces. Our group is engaged in the creation of interface libraries used in many services, including Search. We support four libraries, which in total include 62 blocks.
If you count all desktop and mobile browsers of all versions, it turns out that we have more than 15 support for them. About a year ago we all tested them manually. The tester simply took and clicked on all this in all browsers and looked to see if something went wrong, whether it worked as intended. This led to the fact that the release process is very delayed. Up to the fact that the development and testing took approximately the same time. Many bugs eluded the tester’s eyes or were detected after quite a long time.
')
We decided that it was impossible to continue this way and decided to automate the testing process. We started with static analysis tools. To check the style of the code, we use the jscs tool, written by our colleague Marat Dulin. For static code analysis, the well-known JSHint is used . And for catching regressions in JS, we write unit tests. This to some extent helped to cope with the problem: the analyzers caught quite stupid errors, and the tests allowed to check the functionality of the unit. But with CSS regressions there was a space. Appearance testing was still carried out with the hands and eyes of the tester. We began to look for tools that would help us in automation.
Consider an example. The picture below is a typical block from our standard library. There are various fonts, lights and shadows.
Probably, we could write a unit test, where we would check if a block has certain CSS properties with a certain one. But such an occupation does not seem particularly interesting, and such tests will not be particularly stable. The best way to test pictures is to compare with the standard.
In addition, you need to comply with several conditions. Firstly, it should be possible to test in several browsers at once. After all, the coincidence of the picture with the standard in Firefox does not guarantee at all that IE will have the same correct picture. Secondly, the screenshots of the blocks need to be removed in different states. The same button when pressed may look very different. In addition, it should be possible to take screenshots of individual blocks, rather than the entire page. There may be dynamic content on the page, due to which there will be false positives. Another advantage is that it’s immediately apparent which block on the page has a problem. Thirdly, I would like to store reference screenshots in the repository. This allows you to version them along with the code, maintain multiple versions of the library with different designs, and local storage allows you to increase the testing performance, since the reference screenshot does not need to go to a URL and remove it again. We would like to write the tests ourselves in JavaScript, since we ourselves are all web developers and we know and love this language.
Before writing our tool, we decided to look at the existing ones. The first such tool - Depicted from Google. Its main advantage is that it does not need a test code. You simply feed him two URLs — the reference site and the person being tested — and he himself goes through all the links, takes screenshots and prepares a report. Unfortunately, he does not know how to use the reference repository and take screenshots of individual blocks.
The second such tool is casper.js + phantom.css: a framework for phantomjs headless browser for integration testing and an addon for testing screenshots, respectively. This thing allows you to shoot fragments and test in different states. However, it is very tied to phantom.js, and testing in other browsers is not possible.
The last tool we studied was Huxley from Instagram. He also does not need the code for the tests, he with the help of a special plug-in records all your actions and allows them to play later. But he can only take screenshots of the entire page and runs tests in only one browser at a time.
It is already clear that none of these tools fits all our parameters. In this table you can see what exactly we lacked in each of them.
As a result, we decided to develop our own. We called it gemini - twins.
I will tell you a little about how it works. You describe several block states. For each state, you can specify a list of actions that must be performed to go to it.
Consider a specific example, a button from the library bem-components. She has four states: initial, with hover, pressed and in focus:
This is how the previous abstract scheme will look for a specific button:
She has an initial state that does not require any action. From it you can go to the hover, for which you need to hover the cursor on it. From this state, pressing the LMB, you can go to the pressed state. When we release the LMB, the button will get the focus.
From the schemes go directly to the code. The test for gemini is the usual node.js module, and we import gemini using the usual
The first step is to set the URL from which we will take screenshots.
Next we need to set the region for shooting. This is done using a list of CSS selectors. In the example, there is only one element, but there can be as many of them as you like. The capture area is defined as the minimum rectangle into which all the listed elements fall.
When finished with the setting, you can proceed to capture screenshots. Our first state is the original one (plain). we don’t need to do anything, just take a screenshot with the
For the second state - hovered - we already need to perform a certain action, move the cursor. This can be done in the second argument of the
The next state is the pressed button. Clicking is done with the
The last state is the button in focus: the button pressed in the last example must be released
In principle, this can end the test, but you can still make a small optimization. In each example, we interacted with the same button. And in all examples, the search element is carried out every time. This can be simplified by performing a search once in the
We also need to create a configuration file. In it we set the root URL, from which the relative URLs specified in the tests will be calculated. The second parameter is the URL for the Selenium Grid (since gemini is based on Selenium, the use of the Grid is mandatory). Well, the list of browsers in which we will test. The config looks like this:
A report on the passage of tests can be viewed directly in the console, or create an html-report.
Screenshots are not a replacement for unit tests. It is not necessary to test complex logic with their help.
Use only static data. If you have some kind of dynamic backend, in which for each test you will receive different data that somehow affect the layout, the screenshots will be useless.
The first such service is Sauce Labs , something like a cloudy Selenium Grid. For open source projects it is free. To integrate it with gemini, we need to set two environment variables: the username and the
And instead of Selenium Grid, you need to register the following URL in the config: http://ondemand.saucelabs.com/wd/hub .
If you do not have a dedicated service for testing pages, you will need the SauceConnect utility, which will open a tunnel between your local host and Souce Labs servers.
Another service is the well-known Travis , usually used for continuous integration. To work with it, you need to install several native dependencies. In particular, gemini needs a graphicsmagick. To run the test in gemini, you need to register the
If you want to integrate both services at the same time, Sauce Labs has quite detailed instructions on this topic. For example, see this demo .
The gemini documentation is available at this link , and the tool itself is available on the githaba . There you can also send your wishes and pull-requests, we always welcome them.
The tool continues to evolve. Most recently, we learned how to calculate its coverage of your CSS tests. To do this, you need to specify the
Also, in the near future a version with a software API and a graphical interface will be released.
If you count all desktop and mobile browsers of all versions, it turns out that we have more than 15 support for them. About a year ago we all tested them manually. The tester simply took and clicked on all this in all browsers and looked to see if something went wrong, whether it worked as intended. This led to the fact that the release process is very delayed. Up to the fact that the development and testing took approximately the same time. Many bugs eluded the tester’s eyes or were detected after quite a long time.
')
We decided that it was impossible to continue this way and decided to automate the testing process. We started with static analysis tools. To check the style of the code, we use the jscs tool, written by our colleague Marat Dulin. For static code analysis, the well-known JSHint is used . And for catching regressions in JS, we write unit tests. This to some extent helped to cope with the problem: the analyzers caught quite stupid errors, and the tests allowed to check the functionality of the unit. But with CSS regressions there was a space. Appearance testing was still carried out with the hands and eyes of the tester. We began to look for tools that would help us in automation.
Consider an example. The picture below is a typical block from our standard library. There are various fonts, lights and shadows.
Probably, we could write a unit test, where we would check if a block has certain CSS properties with a certain one. But such an occupation does not seem particularly interesting, and such tests will not be particularly stable. The best way to test pictures is to compare with the standard.
Conditions
In addition, you need to comply with several conditions. Firstly, it should be possible to test in several browsers at once. After all, the coincidence of the picture with the standard in Firefox does not guarantee at all that IE will have the same correct picture. Secondly, the screenshots of the blocks need to be removed in different states. The same button when pressed may look very different. In addition, it should be possible to take screenshots of individual blocks, rather than the entire page. There may be dynamic content on the page, due to which there will be false positives. Another advantage is that it’s immediately apparent which block on the page has a problem. Thirdly, I would like to store reference screenshots in the repository. This allows you to version them along with the code, maintain multiple versions of the library with different designs, and local storage allows you to increase the testing performance, since the reference screenshot does not need to go to a URL and remove it again. We would like to write the tests ourselves in JavaScript, since we ourselves are all web developers and we know and love this language.
Before writing our tool, we decided to look at the existing ones. The first such tool - Depicted from Google. Its main advantage is that it does not need a test code. You simply feed him two URLs — the reference site and the person being tested — and he himself goes through all the links, takes screenshots and prepares a report. Unfortunately, he does not know how to use the reference repository and take screenshots of individual blocks.
The second such tool is casper.js + phantom.css: a framework for phantomjs headless browser for integration testing and an addon for testing screenshots, respectively. This thing allows you to shoot fragments and test in different states. However, it is very tied to phantom.js, and testing in other browsers is not possible.
The last tool we studied was Huxley from Instagram. He also does not need the code for the tests, he with the help of a special plug-in records all your actions and allows them to play later. But he can only take screenshots of the entire page and runs tests in only one browser at a time.
It is already clear that none of these tools fits all our parameters. In this table you can see what exactly we lacked in each of them.
As a result, we decided to develop our own. We called it gemini - twins.
Scheme of work
I will tell you a little about how it works. You describe several block states. For each state, you can specify a list of actions that must be performed to go to it.
Consider a specific example, a button from the library bem-components. She has four states: initial, with hover, pressed and in focus:
This is how the previous abstract scheme will look for a specific button:
She has an initial state that does not require any action. From it you can go to the hover, for which you need to hover the cursor on it. From this state, pressing the LMB, you can go to the pressed state. When we release the LMB, the button will get the focus.
From the schemes go directly to the code. The test for gemini is the usual node.js module, and we import gemini using the usual
require node. First we need to create our test suite. This is done with the gemini.suite command. We transfer the name of the set and the function in which we will further configure this set. All further code that I will give in the examples occurs inside this function. var gemini =require('gemini'); gemini.suite('button', function(suite) { }); The first step is to set the URL from which we will take screenshots.
suite.setURL('/some/url'); Next we need to set the region for shooting. This is done using a list of CSS selectors. In the example, there is only one element, but there can be as many of them as you like. The capture area is defined as the minimum rectangle into which all the listed elements fall.
suite.setCaprureElements('.button'); When finished with the setting, you can proceed to capture screenshots. Our first state is the original one (plain). we don’t need to do anything, just take a screenshot with the
capture command, to which we transfer the name of the state. suite.capture('plain'); For the second state - hovered - we already need to perform a certain action, move the cursor. This can be done in the second argument of the
caprure function. suite.capture('hovered', function(actions) { actions.mouseMove('.button'); }); The next state is the pressed button. Clicking is done with the
mouseDown command. In this example, you can still see an alternative way to set the element: not sending the CSS selector directly, but wrapping it in the find function. Why this is needed and how useful it is, I will tell you a little bit below. suite.capture('pressed', function(actions, find) { actions.mouseDown(find('.button')); }); The last state is the button in focus: the button pressed in the last example must be released
mouseUP command. siute.capture('clicked', function(actions, find) { actions.mouseUp(find('.button')); }); In principle, this can end the test, but you can still make a small optimization. In each example, we interacted with the same button. And in all examples, the search element is carried out every time. This can be simplified by performing a search once in the
before function using the find function, saving the results to a variable. In the future, instead of searching, you can use it. The final version of the test will look like this: var gemini = require('gemini'); gemini.suite('button', function() { suite.setUrl('some/url') .setCaptureElements('.button') .before(function(actions, find) { this.button = find('.button'); }); .capture('plain') .capture('hovered', function(actions, find) { actions.mouseMove(this.button); }) .capture('pressed', function(actions, find){ actions.mouseDown(this.button); }) .capture('clicked', function(actions, find) { actions.mouseUp(this.button); }); ); We also need to create a configuration file. In it we set the root URL, from which the relative URLs specified in the tests will be calculated. The second parameter is the URL for the Selenium Grid (since gemini is based on Selenium, the use of the Grid is mandatory). Well, the list of browsers in which we will test. The config looks like this:
rootUrl: http://localhost:8000 gridUrl: http://localhost:4444/wd/hub browsers: firefox-v30: browserName: firefox version: 30 opera-v12: browserName: opera version: 12 A report on the passage of tests can be viewed directly in the console, or create an html-report.
Useful tips
Screenshots are not a replacement for unit tests. It is not necessary to test complex logic with their help.
Use only static data. If you have some kind of dynamic backend, in which for each test you will receive different data that somehow affect the layout, the screenshots will be useless.
Integration with third-party services
The first such service is Sauce Labs , something like a cloudy Selenium Grid. For open source projects it is free. To integrate it with gemini, we need to set two environment variables: the username and the
access key , which will be given to us after registration. SAUCE_USERNAME=<USERNAME> SAUCE_ACCESS_KEY=<ACCESS KEY> And instead of Selenium Grid, you need to register the following URL in the config: http://ondemand.saucelabs.com/wd/hub .
If you do not have a dedicated service for testing pages, you will need the SauceConnect utility, which will open a tunnel between your local host and Souce Labs servers.
Another service is the well-known Travis , usually used for continuous integration. To work with it, you need to install several native dependencies. In particular, gemini needs a graphicsmagick. To run the test in gemini, you need to register the
gemini test in the package.json of our project: "scripts": { "test": "gemini test" } If you want to integrate both services at the same time, Sauce Labs has quite detailed instructions on this topic. For example, see this demo .
The gemini documentation is available at this link , and the tool itself is available on the githaba . There you can also send your wishes and pull-requests, we always welcome them.
The tool continues to evolve. Most recently, we learned how to calculate its coverage of your CSS tests. To do this, you need to specify the
coverage: true parameter in the config. After running the tests, the report will be in the gemini-coverage folder. The opportunity is still experimental, we welcome your feedback.Also, in the near future a version with a software API and a graphical interface will be released.
Source: https://habr.com/ru/post/238323/
All Articles