Experience in automating regression visual testing in Java + Selenium Webdriver + aShot

Hello.

In this article, I would like to talk about my experience in automating visual regression testing.

Introduction

I decided to do automation of visual testing after I accidentally discovered a small bug in the layout. The project changed the design of one page, and the changes touched non-task pages.

Having decided that manually checking the appearance of the site pages wasn’t an option, I took up automation.
')
We needed a simple solution with convenient reports. To be able to write autotests without any restrictions, compare screenshots of pages and site elements and use different browsers.

After a short search for a ready-made solution / framework, it was decided that it would be easier to write your own solution with the necessary functionality and format of reports. Of the options I saw, only the Galen Framework seemed suitable, but I found it after I wrote my solution.

After implementing and testing autotests, I decided to refine the tests a bit and create a separate project, so that I could use it later on other sites and projects.

about the project

VisualRegressionBoilerplate is a relatively simple project with a specific structure.

Something like a boilerplate template / project for visual autotests.

The project is intended for small sites / projects. For those who need a simple solution, without complex frameworks.

Opportunities

• Comparison of screenshots of site pages. You can specify one or more ignored items.
• Comparison of screenshots of site elements.
• Tests can be run with different screen widths: mobile - 360px, tablet - 768px, desktop - 1920px
• By default, two browsers are available - chrome and firefox + the same browsers in headless mode.
• Report generator - 4 screenshots are created for each test - current, expected, difference, gif. If there are no expected screenshots, actual screenshots will be saved as expected.

Used technologies

• Java
• Testng
• Maven
• Selenium webdriver
• aShot - a library for comparing images

How to start working with a project

1. Install java, maven, browsers.
2. In the constructor of the DriverWrapper class, you can add or remove browsers if necessary
3. The TestConfig class stores all project settings - browser, default browser window size, etc. Here you can add or change settings.
4. In the BasePage class, set the address of the tested site. You can set a different address for different environments (dev, stage, prod). BasePage and the rest of the classes in the app module are an example of the page object pattern. Using it is optional.
5. Examine the TestExample test class and create your own test classes as an example.
6. Add new test classes to testng.xml
7. Run tests with the necessary parameters through maven
8. Check report in report folder

In the README project, everything is described in more detail.

How to write tests

In general, as you like. There are no restrictions. You can use the page object pattern or something else.

All you need is to use the following functions to compare screenshots:

For pages:

Comparison of page screenshots:

comparePageScreenshots("index_page"); 

Comparison of page screenshots with ignoring one element:

 comparePageScreenshots("index_page_ignored_element", IndexPage.MACKBOOK_BLOCK); 

Comparison of page screenshots with ignoring several elements:

  comparePageScreenshots("index_page_ignored_element", new String[]{"section.panel.features.dark", "div.macbook"}); 

Comparison of page element screenshots - element can be searched by css locator

 compareElementScreenshots("index_page_element", IndexPage.FORGE_BLOCK); 

Or you can pass an object of class WebElement

 compareElementScreenshots("index_page_element", driver.findElement(By.cssSelector("a.full.forge"))); 

Possible problems

• Tests may fail due to dynamic content on the page. The problem is solved by ignoring elements with dynamic content.
• Tests may fail due to a few pixels difference. You can fix this by editing the setting - TestConfig.allowableDiffSize .
• Sometimes all the images or some elements on the site do not have time to load. To fix this, I wrote the preparePageForScreenshot () function in which, using javaScript, the page scrolls up and down. But this does not always help.

Summary

At the moment, the project has about 50 tests - pages and different page elements.
Through the bash script autotests are launched and the site is tested in three extensions at once (mobile, tablet, desktop).

There are some problems due to dynamic content and due to the fact that some elements of the site do not always have time to load on time, but autotests can cope with their task.

Any changes to the layout are immediately visible.

Link to the project repository - VisualRegressionBoilerplate