Using Screenshots for Testing

Written at the request of the next topic - How do we test the search in Yandex. Screenshot-based testing of result blocks

A story about deep refactoring.

In our application there was one module implemented at the dawn of business. The module was implemented using patterns different from other modules, exploited global variables in its entirety, consisted of more than hundreds of javascript files that were closely related to each other. The little-known framework, on the basis of which the module was made, was no longer used anywhere. As a result, correcting errors or adding new functionality took a team 2-3 times longer compared to other modules. In connection with future changes in functionality, there is an urgent need to refactor this rather big module.

Unit testing in this case did not fit, because the module changed slightly less than completely. Phantomjs was chosen to automate the functional testing of the module .
The visual component is important in the operation of the application - highlighting of all elements, the correct location on the page, the correct display of data. To track visual changes, it was decided to take screenshots during tests. Good phantomjs makes it easy to do :

page.render('test0001-after-login.png'); 

It was written a sufficient number of tests, covering the functionality in the known volume. As it turned out, we did not know deep enough :) These tests generated several hundred screenshots. Obviously, comparing pictures to each other is a completely ungrateful exercise. You need to do this automatically. In one of the previous projects, I implemented a number of scripts with manipulating pictures on Python (cut, glue, find regions, superimpose pictures on each other, etc.) and was ready to charge it for this task.
Everything went without the invention of bicycles. A powerful cross-platform utility package was used - ImageMagick
compare can compare two screenshots, highlight the difference and display information on the screen.
Call example:

 compare.exe -metric AE shots/test-002_opentree.png reference/test-002_opentree.png diff-002.png 2>&1 

We take a fresh screenshot, compare with the reference, display text information on the screen, and save the graphic to a file for manual viewing of details.
')
It is difficult to notice the difference between these two screenshots:



But when the difference is highlighted, everything is quite obvious - one block has a solid border, not a dotted one:


The first two pictures are made with phatomjs test, and the last one is the result of compare . Some of the information is blurred because I do not have official permission to publish the interface of our program, even if it is outdated.

The final pen stroke is a poweshell script that compares a folder with fresh screenshots and a folder with reference pictures, and after comparing it kills those that are of no interest. The script can easily be converted to a more platform independent language.
Do not forget to look into the script before running it from the command line. Windows by default does not allow running unsigned scripts. What to do with it is written in the cap.
The script is here - detectChanges.ps1

Shl. I apologize for the misinformation about using Python to solve this problem.