Automatic testing of web interfaces in Virto Commerce

In the early stages of development, you can get by manually testing a given test plan. But with the advent of the modular architecture, when several development teams can make their changes simultaneously, there is an exponential increase in the number of test scenarios. Manually chasing them is becoming harder and harder.

In this article we will talk about how we at Virto Commerce automate the testing of important business scenarios.

Why all this?

I will give an example from our experience - a small edit in the CSS-style led to the fact that the “Add product to cart” button was no longer visible on the mobile device. Unfortunately, in this form, it was tested and hit the release. There are many examples of such examples when new functionality, minor changes and corrections break important business scenarios. Unfortunately, we often find out about them after releases and from customers. To avoid this, more than two years ago we began to implement E2E testing as part of the development process. After that, most of the errors were revealed at the development stage, and not in the combat environment.

E2E is testing a business scenario from start to finish. The E2E test simulates the user's real actions in a real browser, precisely as the real user will use the solution.
')
We use E2E tests:

To control the regression
To describe the system
For integration into CI / CD

This includes ensuring that all modules function and work together correctly and predictably.

E2E tests allow us to cover application sections that are not verified by Unit and integration tests. This is due to the fact that Unit tests and integration tests cover individual parts of the application and test the isolated part of the functionality. Even if these parts work well on their own, you cannot be completely sure if they will work together. Thus, having a set of E2E tests on top of the Unit and integration allows us to test the entire solution most fully.

It takes time and resources to write and run E2E tests. Google, for example, offers a 70/20/10 division: 70% unit tests, 20% integration tests, and 10% E2E tests. The exact combination will be different for each project, but in general it should retain the shape of the pyramid.

E2E tests are not simple and at first seem expensive to develop and maintain. In Virto Commerce, we are constantly working to simplify development and reduce the cost of supporting E2E tests when new releases are released. We hope that our solutions will help you to simplify the use of E2E in your projects.

A good user story is an almost complete E2E test.

You can often hear that writing E2E tests takes time and is difficult to maintain. Yes, this is so, they are not easy to maintain in the same way as the documentation. Why not make them part of the development process and writing documentation, thereby fixing the business scenarios made.

Creating an E2E begins with a user story description. As thoroughly as we prepare the description at this stage, it will be as easy for the development team to write an E2E test, which will demonstrate that all systems work correctly when implementing this scenario.

An example of a user history for the add product button to the cart:

GIVEN I am signed in to the storefront AND Some product page is open (/product-name) AND my cart is empty WHEN I click to the "Add" button on the item with the following parameters: THEN I should see a dropdown menu where I can choose from 1 to 10+

which then turns into the following test:

After some time, technical writers or a new development team, instead of reading the documentation, can use the tests in order to run and see the implemented scripts in a real browser or automatically record video tutorials.

It's easy - install and configure Protractor

As a testing tool, we selected Protractor, an open source E2E test automation system designed specifically for AngularJS web applications.

Protractor is a Node.js program built on top of WebDriverJS. Protractor works as a solution integrator that combines powerful technologies such as Node.js, Jasmine, Selenium, Mocha, Cucumber and Web.

Protractor knows about AngularJS, which allows to simplify the writing of tests, without spending attention on waiting for the launch of the AngularJS project, updating the page, etc. ... Experience shows that the threshold for a developer to enter is very low.
Use npm to install Protractor globally:

 npm install -g protractor

webdriver-manager is a utility that allows you to easily configure an instance of Selenium Server. Selenium Server - will be used to transfer commands between the test and the real environment.

Run this command to install or download:

 webdriver-manager update

And run:

 webdriver-manager start

This command will launch Selenium Server and open a window for log output. Protractor will send requests to this server to control the browser.

Protractor is ready to go. A more detailed description of the basic steps for writing tests can be found on the main site .

Proper project structure

E2E tests are necessarily in the same repository, where the application or theme.
We have abandoned the use of external E2E testing services, because, with seeming simplicity, services complicate the launch of tests on a local machine and subsequent maintenance. The code and tests are in different physical locations, which leads to the fact that developers forget to update them.

Finding the application code and test in one repository allows you to simplify the maintenance and support of the project.

A basic example can be found here .

The project creates an E2E folder of the following structure, in which Page Objects are used to organize tests.

 E2E/ |--cart | |--cart.pageObject.js | |--*.spec.js |--home | |--home.pageObject.js | |--*.spec.js |--conf.js

conf.js - the configuration file is in the root
for each object of the interface, a folder is created in which is located
- pageObject file to describe the elements on the page
- several spec files - in which there are tests

The project must use:

baseUrl - which allows you to use the test to test different environments
params - which are used to find key elements on the page or fill out forms

github.com/VirtoCommerce/vc-storefront/blob/master/VirtoCommerce.Storefront.Test/e2e/conf.js#L21

github.com/VirtoCommerce/vc-storefront/blob/master/VirtoCommerce.Storefront.Test/e2e/conf.js#L29

These parameters can then be changed by the IT administrator when setting up startup in Jenkins, in automatic mode for DEV and QA environments.

 protractor conf.js --baseUrl=https://some-url.azurewebsites.net/--params.api.endpoint=https://some-admin-url.azurewebsites.net

Optimization of finding elements on the page

Protractor offers very flexible methods for finding elements on the page:

by.binding
by.model
by.repeater
by.className
by.css
by.select ()
by.partialButtonText ()
...

But in recent projects we have come to the model that the elements that participate in the test are marked with a special class with the prefix protractor-. In the test, these elements are found by the method of.className.

This simplifies test maintenance and modification, as the programmer or automated tools can determine that an element that participates in the E2E test has been modified.

What browsers are we testing

By default, all processes are configured to run tests in the Chrome browser. As our experience of using E2E tests in Chrome shows, it allows us to identify most of the problems and at the same time not complicate the writing of tests.

If you need to test in several browsers, then this is done on the one hand very simple, on the other hand there are difficulties with setting up and the presence of errors in the implementation of drivers for different browsers.

In various client projects, we additionally carried out testing in Firefox, Safari and IE. Firefox tests virtually duplicate results in Chrome. But the launch of E2E in Safari and IE required setting up the system, opening ports, disabling security and editing the registry, but this made it possible to automatically detect a lot of problems with the display and incompatibility of scripts in the application.

To add testing in the browser, you need to download and install the desired WebDriver.

And add a new section multiCapabilities in configuration file:

 exports.config = { … multiCapabilities: [ { 'browserName' : 'chrome' }, { 'browserName' : 'firefox' } ], … };

Since modern web applications support the work at different resolutions, it is necessary to take this into account when writing tests.

Protractor allows you to run tests for different screen resolutions.

For this, there is the option of setting the screen size via browser.driver.manage (). Window (). SetSize. In this example, we set the screen size to 1600 by 800.

 exports.config = { … capabilities: { 'browserName': 'chrome' }, onPrepare: function() { browser.driver.manage().window().setSize(1600, 800); }, … };

Or through the multiCapabilities section in the configuration file. In this example, we will run tests in the Chrome browser, for three different permissions.

 multiCapabilities: [ { 'browserName': 'chrome', 'chromeOptions' : { args: ['--lang=en', '--window-size=1920,1080'] }, specs: ['specs.js'] }, { 'browserName': 'chrome', 'chromeOptions' : { args: ['--lang=en', '--window-size=1680,1050'] }, specs: ['specs.js'] }, { 'browserName': 'chrome', 'chromeOptions' : { args: ['--lang=en', '--window-size=1600,900'] }, specs: ['specs.js'] }]

E2E is part of the documentation.

Once again, E2E is good as an element of documentation, demonstrating what was done by the development team. A new development team or technical writer can run the tests and see the realized scripts live. Therefore, document how a new employee can run these tests.

CI / CD Integration

Any tests should be current and run periodically. If this is not done, then it is not worth spending time writing tests, they will become unusable after a couple of weeks, and it will be easier to rewrite them from scratch.

We have taken an important step in integrating E2E tests into the CI / CD and in becoming an E2E part of the deployment process.

Integration into the CI / CD allows you to run E2E tests automatically for the DEV and QA environments to provide feedback and add insight that the system remains working even after a small change. And if the decision is broken, then give information when and with what change it happened.

First, tests are run when one of the modules is updated and the new version falls on the QA environment.

Secondly, E2E tests are run at night according to a schedule on the DEV and QA environments in automatic mode.

Thirdly, if necessary, the developers themselves can run the tests required.
As a result of the launch, all project participants from the development team to the client receive a letter with an HTML report on the test results.

It is important to note that the availability of information on the test result to the entire project team is mandatory. This gives confidence to the development team on the one hand and manage customer expectations on the other. The development team can make changes faster, while realizing that the core business scenarios are working. And the client receives information about the quality of the project, that most of the problems are identified before the release. And even if the test reveals a problem and the release is postponed, then information appears about what exactly broke and how the process of work on the correction proceeds.

The report version of the plugin protractor-jasmine2-html-reporter (https://github.com/VirtoCommerce/protractor-jasmine2-html-reporter) is used for generating reports. This plugin generates an HTML report and inserts screenshots of inline elements.

Installing it is very simple:

Installing protractor-jasmine2-html-reporter
Adding protractor-jasmine2-html-reporter to the project
Connection report onPrepare method

 var HtmlScreenshotReporter = require('protractor-jasmine2-html-reporter'); var reporter = new HtmlScreenshotReporter(self.options); var name = browser.suite; self.options.reportTitle = name + '-report.html'; jasmine.getEnv().addReporter(reporter);

Continually improve the process.

Constantly work in a team to reduce the cost and increase the speed of development and support of E2E tests.

Conduct training and review of writing tests. See what new components and services can be used.

For example, we are considering the Cucumber connection option. Cucumber is a tool for running automated tests written in simple language.

findings

E2E tests are not so simple and at first seem expensive to maintain, but they are very valuable because they are an indicator of the health of important business scenarios.

From the experience of using E2E tests in the Virto Commerce project, we can draw the following conclusions:

Writing an E2E is part of the development process. And it is very important, as it tests the work of business processes and the entire solution. A huge number of errors were revealed at the DEV and QA stages, and not in the combat environment.
Tests and code should be in the same repository and be accessible to the developer.
The quality of the E2E test depends on the description of the user history, if it is written correctly, it is easier for the development team to create an E2E test.
E2E tests can act as documentation and capture made scripts.
E2E tests should be run continuously and automatically on the DEV and QA environments. If you have not automated the launch, then you should not waste time on tests, they will become unusable in a couple of weeks.
Test results should be available to all project participants. It gives a picture of what is happening.
E2E is not a 100% solution to all problems. Follow the 70/20/10 rule: 70% unit tests, 20% integration tests and 10% E2E tests
Constantly work in a team to reduce the cost and increase the speed of development and support of E2E tests.