Testing the layout of the news site with responsive design
One of the most remarkable tasks that QA-department of EastBanc Technologies ever faced was to create an automated testing system for the site www.washingtonpost.com . This is an electronic newspaper, realized in the form of an information and news portal.
The main reason for the need to create an automated testing system was that the application planned a transition to a new CMS (the so-called PageBuilder), which had to replace several other CMS used before to publish content in various sections of the site. With this kind of migration, it is very important not to make mistakes, so that the content published through the new CMS on different types of pages looks appropriate.
We are not faced with the task of checking all pages for compliance with our tests. Our task is to identify the PageBuilder bugs, check the reliability of the page layout created by the freshly baked PageBuilder, and draw the attention of the editors of the Washingtonpost to the nuances of filling a specific page with content that may lead to potential problems in displaying pages.
The creation of a testing system is under active development, but some points that are interesting in our opinion can already be presented to the general public.
')
Before we do this, it is necessary to note one feature of the project: all testing takes place outside. Those. we, like any other user, use to test the combat version of the site.
Studying the Internet, we stopped at the following approaches and tools. To test the page frame, we adopted the Galen framework, which was later integrated with testNG.
Naturally, the passed Galen-test for the page frame does not mean the validity of the layout. In addition to the location of the blocks, you need to check the display of various elements inside the block. We decided to test the internal contents of the blocks by comparing screenshots.
At the mercy of the screenshot tests, various logos, buttons, some blocks with a specific display fall out - everything that Galen does not reach and that it is difficult / impossible to verify with functional tests.
Azure color - tested by Galen, green filled - screenshot tests:
Caution! Big picture
Galen and Screenshot tests can successfully replace some functional tests, winning at the same time sometimes in clarity, sometimes in speed, and sometimes in both. We choose the testing method for a particular case through a collective discussion of the test case for each type of page based on performance criteria, ease of support, completeness of the test coverage and visibility.
For example, there are 2 blocks for which we initially wrote functional tests: Most Read and Information Block. Now we check the first one with screenshots, and the second with the galen-test.
MostRead Block, screenshot test:
Regarding the functional test: the lines of code have become significantly less, the completeness of the test coverage has increased, and updating the test when the appearance of this block on the page changes does not take long.
Testing of this unit is discussed in the chapter on the screenshot method.
WaPo Information Block:
Galen copes without any problems with checking the conformity of the text and links of this block: the links themselves are set in the locator, and the matching of the text with the internal galen-check. Regarding the functional test, the completeness of the test coverage has not changed, but due to the fact that the checks take place within the framework of one test, we significantly save time.
Galen-test code .
Our automated testing system uses: Java, Maven, TestNG, Selenium WebDriver , Selenium Grid, Galen Framework .
The cross-platform set of utilities ImageMagick actively helps us in creating Screenshot-based tests.
Immediately I would like to note that we write the test code in Java using the PageObject pattern and the Yandex Elements framework, HTML Elements . To run the tests, maven and testNG are used.
To facilitate the launch of tests, viewing the history of test launches, viewing reports without attracting highly qualified specialists, we are developing a separate application - Dashboard.
It will not be superfluous to emphasize that now we are still at the stage of researching how to properly organize the entire testing process, and not all approaches are fully mastered and studied.
Galen Framework has many undoubted advantages: it is a flexible, easy-to-use tool with extensive testing capabilities for adaptive design. In addition, it is well documented and is actively developing at the moment.
Galen Framework has already been described in some detail in one of the articles . If you briefly describe the principle of working with Galen, it looks like this: you write the page specification (the so-called spec file) using a special, well-documented and intuitive syntax. The spec file describes the interposition, size, indents, nesting of page elements and some other parameters and conditions with which the page layout should correspond, you can check even the correspondence of the text inside the element. And all these checks will be applied depending on the tags specified by us.
Tags in the spec-file can be set as follows:
Galen performs all the checks, and then generates a visual report in the form of an html-file. The report indicates which specific checks failed for this test, and for each of the failing checks, you can see a full screenshot of the test page, where elements that did not pass a specific check will be highlighted.
For example, not passed the test for the distance between adjacent elements will look in the report as follows:
When you click on the check highlighted in red, a screenshot of the entire checked page is displayed with the following highlighting of the elements:
Galen Framework accepts the following parameters as input:
As you can see, by varying the parameters supplied by Galen, you can achieve almost complete test coverage for the framework of our site.
After we decided on the site framework testing tool, the next task was to choose a scheme that allows you to provide maximum page coverage with Galen-tests.
And which pages to choose to test the layout, if the test is designed to test a variety of pages of the same type?
We decided not to bother much, each time we started the test suite to select a random page from a subset (i.e., to test a subset of recipe pages, we chose one of the recipes and passed its url to all the tests according to the layouts). Since it’s not worth checking all the pages of the task, the option of choosing a random page seemed optimal. The url of a random page of a subset of pages to be tested is transmitted to Galen by a method common to all tests within our automated site testing system (except for tests for testing layout, we also have functional and screenshot tests).
For example, there are 2 options for displaying the same type of pages - recipe pages, in one of which the layout contains an error:
From the example it can be seen that the block “Most Read”, which should be located in the right column of the page, on the left page is located in the main part, and not on the right. To check the absence of such problems, you need to check a large number of pages and take into account many factors.
First, there was an idea to choose the most common devices and use their resolution to run tests. However, the clearly traced tendency of accelerated mobilization of the planet does not allow us to single out (and, even more, predict) any unconditional leaders in this field. Devices that allow you to view web applications, a great many, and the unification of permissions for such devices is not at all a fashionable task today. Suddenly, the thought that the adaptive design was adaptive in order to be correctly displayed on any valid resolution saved our minds and stopped further research in this area. The decision was made: we test the layout on all valid resolutions.
All resolutions from min Viewport width = 241 px (less the browser does not decrease) to max Viewport width = 1920px (upper limit - by a simple will power) were assigned valid resolutions. We have not yet had a page where the height of the viewport for the purposes of automated testing was the determining parameter, so we do not pay attention to the height.
How to test the layout at all resolutions?
To begin with, the entire range of valid resolutions was divided into ranges of differing layouts. The “rubber” boards themselves, but a different arrangement of the blocks allows a distinction to be made. Determining the boundaries of the layouts is easy - we pull the corner of the browser and look at which boundary point the page blocks change: their relative positions, the number and / or behavior. For simplicity, we take into account only the width of the viewport. The following table turned out:
DESKTOP: max 1920px, min 1018px;
LAPTOP: max 1017px, min 769px;
TABLET: max 768px, min 481px;
MOBILE: max 480px, min 361px;
SMALL_MOBILE: max 360px, min 280px.
By the way, we have so far decided not to test the SMALL_MOBILE layout, since the number of users viewing Washingtonpost on devices with such resolution is catastrophically small (speculative conclusion, and there is no problem to add during testing later). It remains for testing 4 ranges with different layouts.
Below is the code to run the galen test for desktop permissions:
At the start of each test, Galen is supplied with random resolution from the range for the given layout (getRandomResolution (DESKTOP)):
And, actually, the range of permissions is set in this form:
Testing on the random choice of resolution from the valid range and the tested page from a subset of the same type of pages, thus, turns into a probabilistic process. The more often we run - the more different bugs we find. With a single successful passing of the test, we can only say that this particular page on this particular resolution is valid. But after 500 successful runs, we can argue that the layout is mostly viable. Immediately make a reservation that "500 successful runs" is a theoretical assessment, and here you need to look at the content and the number of equivalent pages.
The launch at a random resolution very soon justified itself and immediately revealed one interesting bug that we most likely would have missed when running tests in a fixed resolution.
Let us consider how this approach helps us on the example of testing a recipe page.
The recipe page frame test runs for a resolution range (Viewport width) from 768px to 1017px. Take for example the page: www.washingtonpost.com/pb/recipes/maple-banana-frozen-yogurt/14143
The test on the boundary points of the Laptop layout'a (1017px and 768px) did not produce errors.
However, after we started running the test at random resolution, in about half of the cases, the tests dropped and the screenshots showed that the blocks from the right column were crawling down under the main content.
The correct view:
Caution! Big picture
Layout is broken:
Caution! Big picture
Inspired by the article , we decided to use the screenshot-based testing method. By the way, to test the layout, we initially relied on this method. Those. There was an idea to compare full-size screenshots of the page with a model prepared in advance, replacing all potentially changing elements with plugs (a previously selected arbitrary image is taken as the plugs). These elements included pictures, flash advertisements and text. The idea failed mainly due to the fact that the pages contained many blocks loaded dynamically, as a result of which the physical dimensions of the screenshots taken and the location of the blocks changed from the launch of the test to the launch. In addition, for some time now Chrome has lost the ability to take full-size screenshots, which also created a number of problems.
Screenshot-based tests now we check those individual elements and blocks on the page for which display is important, and / or testing of which by functional or galen tests is difficult or impossible.
For example:
This is what the MostRead block looks like on the washingtonpost.com main page (left) and the model with which we will compare the screenshot of this block (right):
The test code looks like this:
The following directory structure is used to store screenshots: /models/HomePage/firefox/HomePage_thePostMost.png
As you can see from here, for different browsers a model screenshot of the required block is taken.
The shootAndVerify () method finds the path to the model according to the class of the transferred page and the name of the browser in which the test is running.
Looking ahead, let's say - it works quite well, and then we will describe some of the details of the process with the proviso that everything is not completely debugged.
As it turned out, the snapshot of the required block may depend on many factors, such as:
The first problem was that the sizes of the screenshots made differed depending on the OS and browser settings. To make the size of the blocks, and, consequently, the screenshots are the same, you need to run a browser with a constant size. You can change the size of the browser window using the appropriate web driver method: driver.manage (). Window (). SetSize (requiredSize). But this way we set the window size, not the size of the visible area we need - the viewport. Vertical scrollbar, by the way, also affects the size of the viewport, and its thickness also depends on the windows theme, so you need to take this into account. The solution to the problem was the calibration method, which adjusts the size of the viewport to the specified dimensions. After starting the first test, the difference between the width of the window size and the width of the viewport is saved to a special parameter and reused during subsequent launches.
The second problem we encountered was the different display of fonts in browsers due to the anti-aliasing parameters. We tried to solve the problem by installing various browser settings, such as:
layers.acceleration.disabled
gfx.font_rendering.cleartype_params.rendering_mode
gfx.direct2d.disabled
But, unfortunately, it did not help.
In addition, for comparing screenshots the ImageMagick utility uses such a parameter as fuzz, which sets the maximum possible difference between screenshots.
We tried to solve this problem by experimenting with this parameter. A small fuzz coefficient did not solve the problem, since the number of different pixels was very large, due to the fact that there was a lot of text, and a large factor meant that the absence of some elements in the blocks did not affect the passing of the test, and led to potentially missing bugs.
The output was the duplication of all the settings of various browsers on all the virtual machines on which the tests were run, and the duplication of the operating system settings themselves.
For example, a test that checks a block of social buttons in which one of the images did not load.
The links in the report are available:
picture model
screenshot of the test block:
the result of comparing these two images:
CommandException tells us that the compared images differ by 251px:
There are also situations where screenshots do not have the same size. In this case, we get the following report:
Sometimes, for unknown reasons, the elements inside the test block are slightly shifted. For such cases, we compare not with one model, but with a group of models matching the mask, i.e. we can have several models of thePostMost block with such names HomePage_thePostMost.png, HomePage_thePostMost (1) .png, and all models are considered valid. Fortunately, the number of such options is finite, usually not more than 2.
As already mentioned above, the technology stack is used to write and run tests: Java, Maven, TestNG, Selenium, Galen Framework. In addition, test results are sent to graphite. The tests are run directly using Jenkins CIS. We will not stop in detail why such a set was chosen. Briefly describe how it is all interrelated.
Selenium Grid is now deployed locally on four virtual machines running Windows 7, where grid nodes are running, and on a Linux machine running a hub. There are 3 instances of firefox and chrome browsers available on each node. In addition, Jenkins and graphite are also deployed on the Linux machine. Galen tests run in the general test run due to integration with TestNG. For this, the corresponding class was written, allowing the use of the jav'ovy Galen API. When implementing the TestNG interaction with galen, we had some problems that were promptly solved thanks to the interaction with the developer galen. The developer of Galen himself willingly cooperates and regularly releases updates for this tool, which expand its capabilities and make it even more convenient. He himself plans to write documentation on integrating galena with TestNG.
Functional, galen, and screenshot based tests are divided using the appropriate group parameter assigned to Test annotation, and it is possible to launch them separately.
Both approaches - the method of comparing screenshots and testing using the Galen Framework - are applicable for testing page layout. They successfully complement each other. The screenshots comparison method is more applicable when you need to test the display of a particular element or block, for example, the sharing panel on social networks or the main menu in the header. A block can contain many icons inside itself, which in turn can be located inside other icons and elements, or have relative positioning with them.
Using Galen to describe all these small moments is quite time-consuming, but one screenshot per browser solves this problem, and comparing screenshots eliminates the option when we can miss something when describing a spec. Galen, in turn, does an excellent job with the relative position of the blocks and checking the headers and fixed text in them. It has good use when you need to test the layout on the same type of template pages that are not loaded with functional logic, for example, any information portal, as in our case, when almost any page of the site is accessible without authorization, or any other user actions. In addition, galen well solves the problems of cross-browser testing under conditions of adaptive layout of the application.
The main reason for the need to create an automated testing system was that the application planned a transition to a new CMS (the so-called PageBuilder), which had to replace several other CMS used before to publish content in various sections of the site. With this kind of migration, it is very important not to make mistakes, so that the content published through the new CMS on different types of pages looks appropriate.
We are not faced with the task of checking all pages for compliance with our tests. Our task is to identify the PageBuilder bugs, check the reliability of the page layout created by the freshly baked PageBuilder, and draw the attention of the editors of the Washingtonpost to the nuances of filling a specific page with content that may lead to potential problems in displaying pages.
The creation of a testing system is under active development, but some points that are interesting in our opinion can already be presented to the general public.
')
Before we do this, it is necessary to note one feature of the project: all testing takes place outside. Those. we, like any other user, use to test the combat version of the site.
Choice of layout testing tools
Studying the Internet, we stopped at the following approaches and tools. To test the page frame, we adopted the Galen framework, which was later integrated with testNG.
Naturally, the passed Galen-test for the page frame does not mean the validity of the layout. In addition to the location of the blocks, you need to check the display of various elements inside the block. We decided to test the internal contents of the blocks by comparing screenshots.
At the mercy of the screenshot tests, various logos, buttons, some blocks with a specific display fall out - everything that Galen does not reach and that it is difficult / impossible to verify with functional tests.
Azure color - tested by Galen, green filled - screenshot tests:
Caution! Big picture
Hidden text
Galen and Screenshot tests can successfully replace some functional tests, winning at the same time sometimes in clarity, sometimes in speed, and sometimes in both. We choose the testing method for a particular case through a collective discussion of the test case for each type of page based on performance criteria, ease of support, completeness of the test coverage and visibility.
For example, there are 2 blocks for which we initially wrote functional tests: Most Read and Information Block. Now we check the first one with screenshots, and the second with the galen-test.
MostRead Block, screenshot test:
Regarding the functional test: the lines of code have become significantly less, the completeness of the test coverage has increased, and updating the test when the appearance of this block on the page changes does not take long.
Testing of this unit is discussed in the chapter on the screenshot method.
WaPo Information Block:
Galen copes without any problems with checking the conformity of the text and links of this block: the links themselves are set in the locator, and the matching of the text with the internal galen-check. Regarding the functional test, the completeness of the test coverage has not changed, but due to the fact that the checks take place within the framework of one test, we significantly save time.
Galen-test code .
Our automated testing system uses: Java, Maven, TestNG, Selenium WebDriver , Selenium Grid, Galen Framework .
The cross-platform set of utilities ImageMagick actively helps us in creating Screenshot-based tests.
Immediately I would like to note that we write the test code in Java using the PageObject pattern and the Yandex Elements framework, HTML Elements . To run the tests, maven and testNG are used.
To facilitate the launch of tests, viewing the history of test launches, viewing reports without attracting highly qualified specialists, we are developing a separate application - Dashboard.
It will not be superfluous to emphasize that now we are still at the stage of researching how to properly organize the entire testing process, and not all approaches are fully mastered and studied.
Testing with the Galen Framework
Galen Framework has many undoubted advantages: it is a flexible, easy-to-use tool with extensive testing capabilities for adaptive design. In addition, it is well documented and is actively developing at the moment.
Galen Framework has already been described in some detail in one of the articles . If you briefly describe the principle of working with Galen, it looks like this: you write the page specification (the so-called spec file) using a special, well-documented and intuitive syntax. The spec file describes the interposition, size, indents, nesting of page elements and some other parameters and conditions with which the page layout should correspond, you can check even the correspondence of the text inside the element. And all these checks will be applied depending on the tags specified by us.
Tags in the spec-file can be set as follows:
Galen performs all the checks, and then generates a visual report in the form of an html-file. The report indicates which specific checks failed for this test, and for each of the failing checks, you can see a full screenshot of the test page, where elements that did not pass a specific check will be highlighted.
For example, not passed the test for the distance between adjacent elements will look in the report as follows:
When you click on the check highlighted in red, a screenshot of the entire checked page is displayed with the following highlighting of the elements:
Galen Framework accepts the following parameters as input:
- the browser in which the check will take place
- the resolution on which to run the test
- url of the test page
- Javascript file that you need (if you need) to apply on the page being launched before starting the checks on the .spec file (for example, if you need to check the display of the page to the user who is logged on the site)
- the name of the .spec file being run
- tags that need to be applied to file checks .spec (for example: desktop, all, if we are testing a layout for a destkop).
As you can see, by varying the parameters supplied by Galen, you can achieve almost complete test coverage for the framework of our site.
After we decided on the site framework testing tool, the next task was to choose a scheme that allows you to provide maximum page coverage with Galen-tests.
Selecting a test page from a subset of similar pages
And which pages to choose to test the layout, if the test is designed to test a variety of pages of the same type?
We decided not to bother much, each time we started the test suite to select a random page from a subset (i.e., to test a subset of recipe pages, we chose one of the recipes and passed its url to all the tests according to the layouts). Since it’s not worth checking all the pages of the task, the option of choosing a random page seemed optimal. The url of a random page of a subset of pages to be tested is transmitted to Galen by a method common to all tests within our automated site testing system (except for tests for testing layout, we also have functional and screenshot tests).
For example, there are 2 options for displaying the same type of pages - recipe pages, in one of which the layout contains an error:
From the example it can be seen that the block “Most Read”, which should be located in the right column of the page, on the left page is located in the main part, and not on the right. To check the absence of such problems, you need to check a large number of pages and take into account many factors.
What permissions to run tests?
First, there was an idea to choose the most common devices and use their resolution to run tests. However, the clearly traced tendency of accelerated mobilization of the planet does not allow us to single out (and, even more, predict) any unconditional leaders in this field. Devices that allow you to view web applications, a great many, and the unification of permissions for such devices is not at all a fashionable task today. Suddenly, the thought that the adaptive design was adaptive in order to be correctly displayed on any valid resolution saved our minds and stopped further research in this area. The decision was made: we test the layout on all valid resolutions.
All resolutions from min Viewport width = 241 px (less the browser does not decrease) to max Viewport width = 1920px (upper limit - by a simple will power) were assigned valid resolutions. We have not yet had a page where the height of the viewport for the purposes of automated testing was the determining parameter, so we do not pay attention to the height.
How to test the layout at all resolutions?
To begin with, the entire range of valid resolutions was divided into ranges of differing layouts. The “rubber” boards themselves, but a different arrangement of the blocks allows a distinction to be made. Determining the boundaries of the layouts is easy - we pull the corner of the browser and look at which boundary point the page blocks change: their relative positions, the number and / or behavior. For simplicity, we take into account only the width of the viewport. The following table turned out:
DESKTOP: max 1920px, min 1018px;
LAPTOP: max 1017px, min 769px;
TABLET: max 768px, min 481px;
MOBILE: max 480px, min 361px;
SMALL_MOBILE: max 360px, min 280px.
By the way, we have so far decided not to test the SMALL_MOBILE layout, since the number of users viewing Washingtonpost on devices with such resolution is catastrophically small (speculative conclusion, and there is no problem to add during testing later). It remains for testing 4 ranges with different layouts.
Below is the code to run the galen test for desktop permissions:
Hidden text
The invokeGalenActions method gives Galen all the preconditions in the form digested by this framework:
GalenActionBuilder allows you to easily add the execution of both native Galen-preconditions (.waitForVisible (5, ".pb-f-ad-leaderboard> div> div> div> iframe") and our relatives (.scrollToElement (". Pb- f-ad-flex ")):
@Test(groups = { "Galen" }) @WebTest(value = "Verify that layout of Article page is not broken on desktop screen resolution.", bugs = {"#5599", "#5601", "#5600"}) public void testArticlepageLayoutOnDesktop() throws Exception { GalenActionsBuilder builder = new GalenActionsBuilder() //advertisement frames become visible only if advertisement placeholder is visible .waitForVisible(5, ".pb-f-ad-leaderboard > div> div > div > iframe") .scrollToElement(".pb-f-ad-flex") .waitForVisible(5, ".pb-f-ad-flex > div > div > iframe") .scrollToElement(".pb-f-ad-flex-2") .waitForVisible(5, ".pb-f-ad-flex-2 > div > div > iframe") .scrollToElement(".pb-f-ad-flex-3") .waitForVisible(5, ".pb-f-ad-flex-3 > div > div > iframe") .injectJavascript("/js/scroll_to_top.js") .waitSeconds(2) .check("/article.spec", Arrays.asList("all", "desktop")); invokeGalenActions(ArticlePage.getRandomArticlePage(), builder.build(), getRandomResolution(DESKTOP)); } The invokeGalenActions method gives Galen all the preconditions in the form digested by this framework:
protected void invokeGalenActions(String url, List<GalenPageAction> actions, Dimension... sizes) throws Exception { run(url, actions, recalculateSizes(sizes)); } GalenActionBuilder allows you to easily add the execution of both native Galen-preconditions (.waitForVisible (5, ".pb-f-ad-leaderboard> div> div> div> iframe") and our relatives (.scrollToElement (". Pb- f-ad-flex ")):
public class GalenActionsBuilder { private boolean built; private final List<GalenPageAction> actions = new ArrayList<>(); public GalenActionsBuilder waitFor(Integer seconds, GalenPageActionWait.UntilType type, Locator locator) { checkUsed(); GalenPageActionWait.Until u = new GalenPageActionWait.Until(type, locator); GalenPageActionWait a = new GalenPageActionWait(); a.setTimeout(seconds * 1000); a.setUntilElements(Lists.newArrayList(u)); a.setOriginalCommand("wait " + seconds + "s until " + type.toString() + " " + locator.getLocatorValue()); actions.add(a); return this; } public GalenActionsBuilder waitSeconds(Integer seconds) { checkUsed(); GalenPageActionWait a = new GalenPageActionWait(); a.setTimeout(seconds * 1000); a.setOriginalCommand("wait " + seconds + "s"); actions.add(a); return this; } public GalenActionsBuilder waitForExist(Integer seconds, String cssSelector) { return waitFor(seconds, GalenPageActionWait.UntilType.EXIST, Locator.css(cssSelector)); } public GalenActionsBuilder waitForVisible(Integer seconds, String cssSelector) { return waitFor(seconds, GalenPageActionWait.UntilType.VISIBLE, Locator.css(cssSelector)); } public GalenActionsBuilder waitForHidden(Integer seconds, String cssSelector) { return waitFor(seconds, GalenPageActionWait.UntilType.HIDDEN, Locator.css(cssSelector)); } public GalenActionsBuilder waitForGone(Integer seconds, String cssSelector) { return waitFor(seconds, GalenPageActionWait.UntilType.GONE, Locator.css(cssSelector)); } public GalenActionsBuilder waitForExist(Integer seconds, Locator locator) { return waitFor(seconds, GalenPageActionWait.UntilType.EXIST, locator); } public GalenActionsBuilder waitForVisible(Integer seconds, Locator locator) { return waitFor(seconds, GalenPageActionWait.UntilType.VISIBLE, locator); } public GalenActionsBuilder waitForHidden(Integer seconds, Locator locator) { return waitFor(seconds, GalenPageActionWait.UntilType.HIDDEN, locator); } public GalenActionsBuilder waitForGone(Integer seconds, Locator locator) { return waitFor(seconds, GalenPageActionWait.UntilType.GONE, locator); } public GalenActionsBuilder withCookies(String... cookies) { checkUsed(); GalenPageActionCookie a = new GalenPageActionCookie() .withCookies(cookies); a.setOriginalCommand("cookie " + Joiner.on("; ").join(cookies)); actions.add(a); return this; } public GalenActionsBuilder injectJavascript(String javascriptFilePath) { checkUsed(); GalenPageActionInjectJavascript a = new GalenPageActionInjectJavascript(javascriptFilePath); a.setOriginalCommand("inject " + javascriptFilePath); actions.add(a); return this; } public GalenActionsBuilder runJavascript(String javascriptFilePath) { checkUsed(); GalenPageActionRunJavascript a = new GalenPageActionRunJavascript(javascriptFilePath); actions.add(a); return this; } public GalenActionsBuilder runJavascript(String javascriptFilePath, String jsonArgs) { checkUsed(); GalenPageActionRunJavascript a = new GalenPageActionRunJavascript(javascriptFilePath) .withJsonArguments(jsonArgs); actions.add(a); return this; } public GalenActionsBuilder check(String specFile, List<String> tags) { checkUsed(); GalenPageActionCheck a = new GalenPageActionCheck() .withSpecs(Arrays.asList(specFile)) .withIncludedTags(tags); actions.add(a); return this; } public GalenActionsBuilder resize(int width, int height) { checkUsed(); GalenPageActionResize a = new GalenPageActionResize(width, height); a.setOriginalCommand("resize " + GalenUtils.formatScreenSize(new Dimension(width, height))); actions.add(a); return this; } public GalenActionsBuilder open(String url) { checkUsed(); GalenPageActionOpen a = new GalenPageActionOpen(url); a.setOriginalCommand("open " + url); actions.add(a); return this; } private void checkUsed() { if (built) throw new IllegalStateException("Incorrect builder usage error. build() method has been already called"); } public List<GalenPageAction> build() { built = true; return actions; } public GalenActionsBuilder scrollToElement(String locator) { String content = String.format("jQuery(\"%s\")[0].scrollIntoView(true);", locator); Properties properties = new Properties(); try (InputStream is = getClass().getResourceAsStream("/test.properties")){ properties.load(is); } catch (Exception e) { throw new RuntimeException("I/O Exception during loading configuration", e); } String workDirPath = properties.getProperty("work_dir"); String tempDirPath = workDirPath + "\\temp"; String auxJsFile = String.format("%s\\%s.js", tempDirPath, locator.hashCode()); File tempDir = new File(tempDirPath); tempDir.mkdirs(); try { PrintWriter writer = new PrintWriter(auxJsFile, "UTF-8"); writer.println(content); writer.close(); } catch (Exception e) { throw new RuntimeException("Exception during creating file", e); } injectJavascript(auxJsFile); return this; } } At the start of each test, Galen is supplied with random resolution from the range for the given layout (getRandomResolution (DESKTOP)):
protected Dimension getRandomResolution(Dimension[] d) { return getRandomDimensionBetween(d[0], d[1]); } private Dimension getRandomDimensionBetween(Dimension d1, Dimension d2) { double k = Math.random(); int width = (int) (k * (Math.abs(d1.getWidth() - d2.getWidth()) + 1) + Math.min(d1.getWidth(), d2.getWidth())); int height = (int) (k * (Math.abs(d1.getHeight() - d2.getHeight()) + 1) + Math.min(d1.getHeight(), d2.getHeight())); return new Dimension(width, height); } And, actually, the range of permissions is set in this form:
public static final Dimension[] DESKTOP = {new Dimension(1920, 1080), new Dimension(1018, 1080)}; Testing on the random choice of resolution from the valid range and the tested page from a subset of the same type of pages, thus, turns into a probabilistic process. The more often we run - the more different bugs we find. With a single successful passing of the test, we can only say that this particular page on this particular resolution is valid. But after 500 successful runs, we can argue that the layout is mostly viable. Immediately make a reservation that "500 successful runs" is a theoretical assessment, and here you need to look at the content and the number of equivalent pages.
The launch at a random resolution very soon justified itself and immediately revealed one interesting bug that we most likely would have missed when running tests in a fixed resolution.
Let us consider how this approach helps us on the example of testing a recipe page.
The recipe page frame test runs for a resolution range (Viewport width) from 768px to 1017px. Take for example the page: www.washingtonpost.com/pb/recipes/maple-banana-frozen-yogurt/14143
The test on the boundary points of the Laptop layout'a (1017px and 768px) did not produce errors.
However, after we started running the test at random resolution, in about half of the cases, the tests dropped and the screenshots showed that the blocks from the right column were crawling down under the main content.
The correct view:
Caution! Big picture
Hidden text
Layout is broken:
Caution! Big picture
Hidden text
Screenshot-based testing method
Inspired by the article , we decided to use the screenshot-based testing method. By the way, to test the layout, we initially relied on this method. Those. There was an idea to compare full-size screenshots of the page with a model prepared in advance, replacing all potentially changing elements with plugs (a previously selected arbitrary image is taken as the plugs). These elements included pictures, flash advertisements and text. The idea failed mainly due to the fact that the pages contained many blocks loaded dynamically, as a result of which the physical dimensions of the screenshots taken and the location of the blocks changed from the launch of the test to the launch. In addition, for some time now Chrome has lost the ability to take full-size screenshots, which also created a number of problems.
Screenshot-based tests now we check those individual elements and blocks on the page for which display is important, and / or testing of which by functional or galen tests is difficult or impossible.
For example:
This is what the MostRead block looks like on the washingtonpost.com main page (left) and the model with which we will compare the screenshot of this block (right):
The test code looks like this:
@Test(groups = { "ScreenshotBased" }) @WebTest("Verifies that 'Post Most' block is displayed properly") public void testMakeupForPostMost() { HomePage page = new HomePage().open(); page.preparePostMostForScreenshot(); screenshotHelper.shootAndVerify(page, page.thePostMost, "_thePostMost"); } The following directory structure is used to store screenshots: /models/HomePage/firefox/HomePage_thePostMost.png
As you can see from here, for different browsers a model screenshot of the required block is taken.
The shootAndVerify () method finds the path to the model according to the class of the transferred page and the name of the browser in which the test is running.
Looking ahead, let's say - it works quite well, and then we will describe some of the details of the process with the proviso that everything is not completely debugged.
As it turned out, the snapshot of the required block may depend on many factors, such as:
- operating system version
- operating system theme
- browser and its version
- various font smoothing options and hardware acceleration.
The first problem was that the sizes of the screenshots made differed depending on the OS and browser settings. To make the size of the blocks, and, consequently, the screenshots are the same, you need to run a browser with a constant size. You can change the size of the browser window using the appropriate web driver method: driver.manage (). Window (). SetSize (requiredSize). But this way we set the window size, not the size of the visible area we need - the viewport. Vertical scrollbar, by the way, also affects the size of the viewport, and its thickness also depends on the windows theme, so you need to take this into account. The solution to the problem was the calibration method, which adjusts the size of the viewport to the specified dimensions. After starting the first test, the difference between the width of the window size and the width of the viewport is saved to a special parameter and reused during subsequent launches.
The second problem we encountered was the different display of fonts in browsers due to the anti-aliasing parameters. We tried to solve the problem by installing various browser settings, such as:
layers.acceleration.disabled
gfx.font_rendering.cleartype_params.rendering_mode
gfx.direct2d.disabled
But, unfortunately, it did not help.
In addition, for comparing screenshots the ImageMagick utility uses such a parameter as fuzz, which sets the maximum possible difference between screenshots.
We tried to solve this problem by experimenting with this parameter. A small fuzz coefficient did not solve the problem, since the number of different pixels was very large, due to the fact that there was a lot of text, and a large factor meant that the absence of some elements in the blocks did not affect the passing of the test, and led to potentially missing bugs.
The output was the duplication of all the settings of various browsers on all the virtual machines on which the tests were run, and the duplication of the operating system settings themselves.
For example, a test that checks a block of social buttons in which one of the images did not load.
The links in the report are available:
picture model
screenshot of the test block:
the result of comparing these two images:
CommandException tells us that the compared images differ by 251px:
There are also situations where screenshots do not have the same size. In this case, we get the following report:
Sometimes, for unknown reasons, the elements inside the test block are slightly shifted. For such cases, we compare not with one model, but with a group of models matching the mask, i.e. we can have several models of thePostMost block with such names HomePage_thePostMost.png, HomePage_thePostMost (1) .png, and all models are considered valid. Fortunately, the number of such options is finite, usually not more than 2.
Technical aspects
As already mentioned above, the technology stack is used to write and run tests: Java, Maven, TestNG, Selenium, Galen Framework. In addition, test results are sent to graphite. The tests are run directly using Jenkins CIS. We will not stop in detail why such a set was chosen. Briefly describe how it is all interrelated.
Selenium Grid is now deployed locally on four virtual machines running Windows 7, where grid nodes are running, and on a Linux machine running a hub. There are 3 instances of firefox and chrome browsers available on each node. In addition, Jenkins and graphite are also deployed on the Linux machine. Galen tests run in the general test run due to integration with TestNG. For this, the corresponding class was written, allowing the use of the jav'ovy Galen API. When implementing the TestNG interaction with galen, we had some problems that were promptly solved thanks to the interaction with the developer galen. The developer of Galen himself willingly cooperates and regularly releases updates for this tool, which expand its capabilities and make it even more convenient. He himself plans to write documentation on integrating galena with TestNG.
Functional, galen, and screenshot based tests are divided using the appropriate group parameter assigned to Test annotation, and it is possible to launch them separately.
Our conclusions
Both approaches - the method of comparing screenshots and testing using the Galen Framework - are applicable for testing page layout. They successfully complement each other. The screenshots comparison method is more applicable when you need to test the display of a particular element or block, for example, the sharing panel on social networks or the main menu in the header. A block can contain many icons inside itself, which in turn can be located inside other icons and elements, or have relative positioning with them.
Using Galen to describe all these small moments is quite time-consuming, but one screenshot per browser solves this problem, and comparing screenshots eliminates the option when we can miss something when describing a spec. Galen, in turn, does an excellent job with the relative position of the blocks and checking the headers and fixed text in them. It has good use when you need to test the layout on the same type of template pages that are not loaded with functional logic, for example, any information portal, as in our case, when almost any page of the site is accessible without authorization, or any other user actions. In addition, galen well solves the problems of cross-browser testing under conditions of adaptive layout of the application.
Source: https://habr.com/ru/post/230291/
All Articles