Testing for Visual Regression Layouts with PhantomCSS
Working with someone else's code is one of the common and difficult problems that I have had to face in my work. In almost every case, the previous developer did not write the code the way I would like.
And such situations occur quite often. Not every client has the need, desire or budget to rewrite the entire project from scratch.
Recently, our team received the code from a new client, and after a small refactoring we were instructed to quickly move on to implementing a new functionality. We understood that we could improve the code by transferring client styles to Sass, and this would simplify our support in the future.
')
On the other hand, we could just rename the files and include them into one precompiled css (without any refactoring). But to improve the code it would be good to refactor styles. This work is more costly, but in the future it would pay for itself. And, most importantly, it would allow us to work faster, with greater confidence that we will not break something.
I used to consider such changes as big risks . In the end, C in CSS is cascading, where order is absolutely important. Restructuring multiple styles means changing the order, which naturally leads to a big risk of breaking something.
As a result, we had to either additionally test our changes manually, or issue an invoice to the customer, who would immediately scare him away with his own cost.
This time it was decided to build a visually regression test suite.
This type of test has recently begun to gain popularity, and this is fully justified. In fact, this is a series of tests that run all over your site and take screenshots of various components, compare them with basic screenshots and warn you about changes.
This may seem a little illogical. We change CSS, because we ourselves want our interface to look different. Simply put, why do we need to build some kind of process that will notify us every time that we broke something when we change something in our styles?
If you change your client styles or work in a team, it is quite easy to make changes to CSS, which you think has affected only 1 component. And only later do you find that your changes have broken something on another page.
To understand the usefulness of visual regression tests, it is important to understand what a person does poorly with.
In fact, a person can really very poorly distinguish changes in visual images. This is due to its physiological and psychological characteristics.
There is even a game based on this. Do you remember the pictures from the “find a few differences” series?
There are a number of real problems that psychologists are trying to solve. On the other hand, scientists have already gained a lot of knowledge in this area, which is applicable to web development.
One important phenomenon that needs to be considered here is the blindness to change.
Research in this area began in 1970. In 1996, George McConkie and Christopher Currie at the University of Urbana-Champaign Illinois conducted a series of studies that generated considerable interest in this area.
Blindness to change is a lack of perception. Experiments have shown that serious changes in the field of view often go unnoticed, regardless of whether they occur gradually, in sudden short flashes, or appear unexpectedly at different time intervals. It is not associated with any visual defect, it is pure psychology.
The McConkie & Currie study found that in some cases one fifth of all changes may go unnoticed. This video is a great example of how many changes can be missed if you don’t focus on them.
There is a wide range of tools for creating tests. I always recommend comparing tools and determining which are best suited for specific tasks.
I chose PhantomCSS as a tool for visual testing of regressions. There were several reasons for this.
First, it has a relatively large and active community on github. When it comes to open source, I always check that a tool or library is being actively developed. Working with frozen open source projects can quickly become a burden.
Secondly, PhantomCSS has a convenient Grunt plugin that allows it to easily integrate into my existing development process.
The PhantomCSS core is a combination of three key components:
And, as I mentioned, we will use Grunt to run our tests.
Now that we’ve figured out what’s what, let's go through all the steps for implementing visual regression testing.
First, we will need Grunt to run the tests, so make sure that it is installed. To do this, at the command prompt, type
Then open the project's Gruntfile, load the PhantomCSS task, and add it to grunt.initConfig ():
I like to use Sass MQ to work with various devices. This approach has an additional advantage: it returns a list of all my devices that I can easily use for my tests.
With PhantomCSS, you can control the width of the browser within your test, but I prefer to abstract from my tests to get more flexibility for tests. Therefore, I delegate this task to Grunt.
With grunt-phantomcss, we can define a set of tests to run on different devices and, as an added bonus, set up saving them in different folders.
In order to maintain semantics and order, I try to name subtasks according to the devices in Sass MQ.
Example:
We conducted the same set of tests that works on different devices and stores the results in the appropriate directories.
In the definition of Grunt you can see that we start the process from the file tests / phantomcss / start.js. This file launches Casper (which then runs the test scripts and our browser), and it looks like this:
Now back to our Grunt. You probably already noticed that we run all the files in the `tests / phantomcss /` directory, which end in `-test.js`. Grunt cycles through each of these files in alphabetical order.
You can decide how to organize the test files. Personally, I create a test file for each component in my application.
After you have implemented your `start.js` file, it is time to write your first test. We will name this file `header-test.js`.
At the beginning of the file, we tell Casper to open the root URL, then take a screenshot of the entire .site-header element. The second parameter is the name of the screenshot. I prefer to call screenshots as well as the component they capture. So much easier to understand and share the results with teammates.
For the simplest example of this test will be enough. However, we can build a more reliable test that covers the actual state of the element, page, and application as a whole.
Casper allows you to automate the interaction that occurs during the operation of the PhantomCSS browser. For example, we could write a check on the state of the button when you hover the mouse as follows:
You can also test the login / logout state. In the `start.js` file, we can write a function that fills the login form in WordPress as soon as we launch the Casper instance.
You may notice that we do this in casper.start () instead of doing it inside each individual test. This session setup inside casper.start () in the `start.js` file makes the session accessible from other files with your tests, since casper.start () always starts first.
For more information, I recommend taking a look at the Casper documentation.
Now that we have implemented the test suite, it's time to run them. At the command prompt, run `$ grunt phantomcss`.
PhantomCSS will automatically take screenshots at the first launch, and set them as basic for comparison with future results.
If the test fails, PhantomCSS will save three different screenshots in a customized folder, and name the files as, `.diff.png` and` .fail.png`.
For example, we changed the font size of the text on one page, but accidentally reduced the font size to another. PhantomCSS will take screenshots for comparison:
Building a set of visual regression tests, of course, is not without problems. The two main problems encountered are dynamic content and the distribution of tests within a team.
The first question is how to handle dynamic content. A set of tests passes through each page, takes screenshots, and compares them. If the page content has dynamically changed, the test will not be able to detect it.
If you work in a team, the changes will always be tested in your own local environment. Testing on one common environment does not always fix the problem, because the content there may still change, for example, if there is a random set of related articles on the page.
To solve this problem, there are two approaches that I like.
The first approach, my favorite, is to use JavaScript to replace the contents inside the elements under test.
Since these tests will not be deployed on a production server, you do not have to worry about XSS vulnerabilities. Thus, before taking a screenshot, I use `.html ()` in my tests to replace a dynamic contact with a static contact, which is taken from the JSON object that I included in my repository.
The second approach is to use a tool called Hologram or mdcss , which allows you to use CSS comments to create automatically generated styles. This approach makes major changes to the workflow, which requires more time and money, but has the added advantage of creating excellent documentation for front-end web components.
The second major problem I encountered during regression testing is how to best distribute these tests among a team of engineers. Until now, in our tests we have rigidly registered our test URL, and this will cause problems when working in a team, because the same URL cannot be used by all programmers for local testing.
To solve this problem, we registered our `$ grunt test` task, which accepts a` --url` parameter, which is then saved to a file locally using grunt.log.
Then, at the beginning of the test file we register:
When started, your tests will look at the `.local_url` file, but if the file does not exist, it will use the default value of` http: // local.wordpress.dev`.
There are many benefits that visually regression testing can bring to your projects. Sprints and continuous integration are increasingly used by developers.
Visual regression testing is also great for working with people on open source projects. In fact, the WordPress project is working on a template library with a built-in regression testing package. This test suite provides a framework that allows a WordPress project to confidently move forward and implement its style refactoring plans.
PhantomCSS is not the only tool. I chose him simply because I considered him for the purposes of our team, he would be the best. If you are interested in visual regression testing, but you didn’t like PhantomCSS, I advise you to take a look at the following tools:
And such situations occur quite often. Not every client has the need, desire or budget to rewrite the entire project from scratch.
Recently, our team received the code from a new client, and after a small refactoring we were instructed to quickly move on to implementing a new functionality. We understood that we could improve the code by transferring client styles to Sass, and this would simplify our support in the future.
')
On the other hand, we could just rename the files and include them into one precompiled css (without any refactoring). But to improve the code it would be good to refactor styles. This work is more costly, but in the future it would pay for itself. And, most importantly, it would allow us to work faster, with greater confidence that we will not break something.
I used to consider such changes as big risks . In the end, C in CSS is cascading, where order is absolutely important. Restructuring multiple styles means changing the order, which naturally leads to a big risk of breaking something.
As a result, we had to either additionally test our changes manually, or issue an invoice to the customer, who would immediately scare him away with his own cost.
This time it was decided to build a visually regression test suite.
This type of test has recently begun to gain popularity, and this is fully justified. In fact, this is a series of tests that run all over your site and take screenshots of various components, compare them with basic screenshots and warn you about changes.
This may seem a little illogical. We change CSS, because we ourselves want our interface to look different. Simply put, why do we need to build some kind of process that will notify us every time that we broke something when we change something in our styles?
If you change your client styles or work in a team, it is quite easy to make changes to CSS, which you think has affected only 1 component. And only later do you find that your changes have broken something on another page.
To understand the usefulness of visual regression tests, it is important to understand what a person does poorly with.
Man vs. Machine
In fact, a person can really very poorly distinguish changes in visual images. This is due to its physiological and psychological characteristics.
There is even a game based on this. Do you remember the pictures from the “find a few differences” series?
There are a number of real problems that psychologists are trying to solve. On the other hand, scientists have already gained a lot of knowledge in this area, which is applicable to web development.
One important phenomenon that needs to be considered here is the blindness to change.
Blindness to change
Research in this area began in 1970. In 1996, George McConkie and Christopher Currie at the University of Urbana-Champaign Illinois conducted a series of studies that generated considerable interest in this area.
Blindness to change is a lack of perception. Experiments have shown that serious changes in the field of view often go unnoticed, regardless of whether they occur gradually, in sudden short flashes, or appear unexpectedly at different time intervals. It is not associated with any visual defect, it is pure psychology.
The McConkie & Currie study found that in some cases one fifth of all changes may go unnoticed. This video is a great example of how many changes can be missed if you don’t focus on them.
Instruments
There is a wide range of tools for creating tests. I always recommend comparing tools and determining which are best suited for specific tasks.
I chose PhantomCSS as a tool for visual testing of regressions. There were several reasons for this.
First, it has a relatively large and active community on github. When it comes to open source, I always check that a tool or library is being actively developed. Working with frozen open source projects can quickly become a burden.
Secondly, PhantomCSS has a convenient Grunt plugin that allows it to easily integrate into my existing development process.
The PhantomCSS core is a combination of three key components:
- PhantomJS or SlimerJS is a primitive browser. PhantomJS is a primitive version of WebKit, while Slimer is a Gecko engine used in Firefox.
- CasperJS - Casper is a JavaScript utility for navigation and testing. It allows us to define a set of actions that occur within our primitive browser.
- ResembleJS is a JavaScript / HTML5 library for image comparisons. She will look for the difference between the results of our tests and the baseline results, and warn you about the differences between them.
And, as I mentioned, we will use Grunt to run our tests.
Implementation
Now that we’ve figured out what’s what, let's go through all the steps for implementing visual regression testing.
Grunt setup
First, we will need Grunt to run the tests, so make sure that it is installed. To do this, at the command prompt, type
$ cd /path/to/your-site Then open the project's Gruntfile, load the PhantomCSS task, and add it to grunt.initConfig ():
grunt.loadNpmTasks('@micahgodbolt/grunt-phantomcss'); grunt.initConfig({ phantomcss: { desktop: { options: { screenshots: 'baselines/desktop', results: 'results/desktop', viewportSize: [1280, 800] }, src: [ 'tests/phantomcss/start.js', 'tests/phantomcss/*-test.js' ] } } }); Testing on various devices
I like to use Sass MQ to work with various devices. This approach has an additional advantage: it returns a list of all my devices that I can easily use for my tests.
With PhantomCSS, you can control the width of the browser within your test, but I prefer to abstract from my tests to get more flexibility for tests. Therefore, I delegate this task to Grunt.
With grunt-phantomcss, we can define a set of tests to run on different devices and, as an added bonus, set up saving them in different folders.
In order to maintain semantics and order, I try to name subtasks according to the devices in Sass MQ.
Example:
grunt.initConfig( { pkg: grunt.file.readJSON('package.json'), phantomcss: { desktop: { options: { screenshots: 'baselines/desktop', results: 'results/desktop', viewportSize: [1024, 768] }, src: [ 'tests/phantomcss/start.js', 'tests/phantomcss/*-test.js' ] }, mobile: { options: { screenshots: 'baselines/mobile', results: 'results/mobile', viewportSize: [320, 480] }, src: [ 'tests/phantomcss/start.js', 'test/phantomcss/*-test.js' ] } } }); We conducted the same set of tests that works on different devices and stores the results in the appropriate directories.
Test Setup
In the definition of Grunt you can see that we start the process from the file tests / phantomcss / start.js. This file launches Casper (which then runs the test scripts and our browser), and it looks like this:
phantom.casperTest = true; casper.start(); Now back to our Grunt. You probably already noticed that we run all the files in the `tests / phantomcss /` directory, which end in `-test.js`. Grunt cycles through each of these files in alphabetical order.
You can decide how to organize the test files. Personally, I create a test file for each component in my application.
We write the first test
After you have implemented your `start.js` file, it is time to write your first test. We will name this file `header-test.js`.
casper.thenOpen('http://mysite.dev/') .then(function() { phantomcss.screenshot('.site-header', 'site-header'); }); At the beginning of the file, we tell Casper to open the root URL, then take a screenshot of the entire .site-header element. The second parameter is the name of the screenshot. I prefer to call screenshots as well as the component they capture. So much easier to understand and share the results with teammates.
For the simplest example of this test will be enough. However, we can build a more reliable test that covers the actual state of the element, page, and application as a whole.
Script Interactions
Casper allows you to automate the interaction that occurs during the operation of the PhantomCSS browser. For example, we could write a check on the state of the button when you hover the mouse as follows:
casper.then(function() { this.mouse.move('.button'); phantomcss.screenshot('.button'); }); You can also test the login / logout state. In the `start.js` file, we can write a function that fills the login form in WordPress as soon as we launch the Casper instance.
casper.start('http://default.wordpress.dev/wp-admin/', function() { this.fill('form#loginform', { 'log': 'admin', 'pwd': 'password' }, true); this.click('#wp-submit'); console.log('Logging in...'); }); You may notice that we do this in casper.start () instead of doing it inside each individual test. This session setup inside casper.start () in the `start.js` file makes the session accessible from other files with your tests, since casper.start () always starts first.
For more information, I recommend taking a look at the Casper documentation.
Running tests
Now that we have implemented the test suite, it's time to run them. At the command prompt, run `$ grunt phantomcss`.
PhantomCSS will automatically take screenshots at the first launch, and set them as basic for comparison with future results.
If the test fails, PhantomCSS will save three different screenshots in a customized folder, and name the files as, `.diff.png` and` .fail.png`.
For example, we changed the font size of the text on one page, but accidentally reduced the font size to another. PhantomCSS will take screenshots for comparison:
Problems
Building a set of visual regression tests, of course, is not without problems. The two main problems encountered are dynamic content and the distribution of tests within a team.
Dynamic content
The first question is how to handle dynamic content. A set of tests passes through each page, takes screenshots, and compares them. If the page content has dynamically changed, the test will not be able to detect it.
If you work in a team, the changes will always be tested in your own local environment. Testing on one common environment does not always fix the problem, because the content there may still change, for example, if there is a random set of related articles on the page.
To solve this problem, there are two approaches that I like.
The first approach, my favorite, is to use JavaScript to replace the contents inside the elements under test.
Since these tests will not be deployed on a production server, you do not have to worry about XSS vulnerabilities. Thus, before taking a screenshot, I use `.html ()` in my tests to replace a dynamic contact with a static contact, which is taken from the JSON object that I included in my repository.
The second approach is to use a tool called Hologram or mdcss , which allows you to use CSS comments to create automatically generated styles. This approach makes major changes to the workflow, which requires more time and money, but has the added advantage of creating excellent documentation for front-end web components.
Distribution of tests within the team
The second major problem I encountered during regression testing is how to best distribute these tests among a team of engineers. Until now, in our tests we have rigidly registered our test URL, and this will cause problems when working in a team, because the same URL cannot be used by all programmers for local testing.
To solve this problem, we registered our `$ grunt test` task, which accepts a` --url` parameter, which is then saved to a file locally using grunt.log.
// All a variable to be passed, eg. --url=http://test.dev var localURL = grunt.option( 'url' ); /** * Register a custom task to save the local URL, which is then read by the PhantomCSS test file. * This file is saved so that "grunt test" can then be run in the future without passing your local URL each time. * * Note: Make sure test/visual/.local_url is added to your .gitignore * * Props to Zack Rothauser for this approach. */ grunt.registerTask('test', 'Runs PhantomCSS and stores the --url parameter', function() { if (localURL) { grunt.log.writeln( 'Local URL: ' + localURL ); grunt.file.write( 'test/visual/.local_url', localURL ); } grunt.task.run(['phantomcss']); }); Then, at the beginning of the test file we register:
var fs = require('fs'), siteURL; try { siteURL = fs.read( 'test/visual/.local_url' ); } catch(err) { siteURL = (typeof siteURL === 'undefined') ? 'http://local.wordpress.dev' : siteURL; } casper.thenOpen(siteURL + '/path/to/template'); When started, your tests will look at the `.local_url` file, but if the file does not exist, it will use the default value of` http: // local.wordpress.dev`.
Finally
There are many benefits that visually regression testing can bring to your projects. Sprints and continuous integration are increasingly used by developers.
Visual regression testing is also great for working with people on open source projects. In fact, the WordPress project is working on a template library with a built-in regression testing package. This test suite provides a framework that allows a WordPress project to confidently move forward and implement its style refactoring plans.
Alternatives
PhantomCSS is not the only tool. I chose him simply because I considered him for the purposes of our team, he would be the best. If you are interested in visual regression testing, but you didn’t like PhantomCSS, I advise you to take a look at the following tools:
Source: https://habr.com/ru/post/271379/
All Articles