A translation of the article was prepared specifically for students of the Python QA Engineer course .

Cross-browser testing is exactly the type of testing that requires a lot of effort and time. Testing a web application in various browsers, on different operating systems, devices, screen resolutions in order to evaluate the rendering of your network content to a wide audience is a serious task. Especially if you do it manually. Automated cross-browser testing with Selenium will save time on routine activities and reduce regression testing time. However, people do not like change. If manual testing is popular in your organization, management will probably have questions when you propose to implement test automation.

Test automation, although extremely profitable, is potentially costly. Is the game worth the candle? This is one of the issues that you may encounter when convincing top management. As you develop your web application, you will need to provide an effective return on investment (ROI) from testing automation with Selenium, emphasizing the benefits of such automation for cross-browser testing of web applications, namely faster work and less manual work.

In this article, we will discuss various metrics for evaluating ROI from testing automation with Selenium, as well as methods for calculating ROI, starting with the basics and ending with advanced methods.

Metrics for evaluating ROI from test automation using Selenium

You, as well as members of your team, can select specific indicators and measurements that will help you analyze ROI from test automation with Selenium, as you plan to start automated testing of your web application from scratch. Performance may vary by organization. Why? This is a matter of priorities, because there are various indicators, such as the number of detected defects (errors), time spent and test coverage, which directly affect the risks, cost, quality and delivery schedule of your product. Some organizations may give priority to the number of defects found since they believe that the quantity is flowing into quality. Some people think differently, because quality is fundamental to them. What do you choose? What do you think is more important: quality or quantity? You can write your thoughts in the comments.

Well, now defining the key metrics for calculating the ROI from test automation with Selenium will be crucial for you, as you need to identify them before you engage in a discussion with senior management.

Test Automation Area with Selenium

It is well known that testing cannot be fully automated. How much you can automate cross-browser testing is a difficult question. If you decide to implement automated cross-browser testing for your web application, you will have to decide on the priorities, that is, which browsers and operating systems should cover the tests. In any case, you cannot cover all the scenarios. The total number of possible scenarios can be in the hundreds or thousands of tests. If your automation script takes a lot of time, it will inevitably affect the worse for the total amount of time you spend every day testing your web application or website.

In short, you have to compare the total number of automated test cases and the number of test cases in which you can apply automation.

In case you want to reduce the execution time of complex test suites, you can use parallel testing with the Selenium Grid . This way you can run multiple test cases at the same time. However, here you have to think about how many simultaneous or parallel sessions will be enough to meet the given requirements. You can calculate this with the concurrency calculator .

Improving this indicator means that teams can quickly find the error and fix it, which leads to lower risks and increased ROI from testing automation with Selenium.

How much time will you save?

Agile is characterized by weekly and two-week sprints and frequent changes in requirements. In such circumstances, the importance of regression testing is increasing. The introduction of automated regression tests will reduce the time required to conduct such testing. This will increase the amount of time to develop or develop another sprint. Saving time is a priority for every organization, especially for startups that need to quickly scale their web application. Is time one of your interests in evaluating ROI from test automation?

The bandwidth of your resources

We know that testing automation with Selenium will help you quickly launch your web application to the market. However, no organization would prefer its employees to sit idle most of the time, waiting for the script to work out. Calculation of ROI requires a thorough analysis of the efforts made by each tester in your team.

Investment budget of resources and tools

Test automation saves time and effort. However, this implies a compromise with pricing. You need to think about how much budget you can spend on maintaining the tools for conducting automated testing.

Total number of defects

The total number of defects detected after each cycle of regression testing gives an idea of ​​the quality of the product and how effective automated testing is for this particular product.

Finding the Right ROI for Automated Testing

Depending on the number of regression cycles that must be performed during the project life cycle, the ROI can take a positive value. Typically, the ROI is calculated using the following formula:

ROI = (   –   ) /    

However, with the advent of Agile and DevOps, the classic method no longer works. In addition, such a measurement is unrealistic, since the number of manual tests cannot be the same as the number of automated ones. Calculating the correct ROI from test automation in the case of Selenium based on quantity is not preferable, but it is still used.

Quality of defects

This, in my opinion, is a very important indicator for calculating the ROI from testing automation with Selenium. I believe that the whole point of testing automation is not to remove the need for manual testers in the project. The purpose of automation is to reduce the time spent by the tester on conducting tests, in the conditions of his tight schedule, so that the tester can come up with more ready-made test cases. Improving the quality of test cases will certainly help improve your web application.

Common errors in calculating ROI from automated testing

Although ROI calculation involves a basic calculation using a simple formula, errors may occur if you miss some important parameters. Let's discuss some common mistakes that people make when calculating ROI.

a source

You do not completely ignore manual testing, do you?

One of the most common mistakes is to use only automated test metrics. Manual testing will always be of particular importance. When we talk about cross-browser testing, we understand that some scenarios can be automated, but there are times when you need to interact with a web application in real time, doing cross-browser testing manually. Because visual defects, for example, are easier to detect manually than running an automated script. Things such as the attractiveness of the appearance of the site when displayed in various browsers or the correct operation of the navigation menu are always checked manually. If you try to automate tests of this kind, they will not bring an increase in ROI. Even if you do not reckon with “manual labor”, you still have to consider the time and money allotted for this.

Imagine the big picture

When measuring ROI from test automation with Selenium, you need to consider a longer period of time. Testing how a particular testing methodology benefits the organization in a short time does not always provide a complete picture. You should find out how it will affect the organization and the team in the long run. Instead of taking months, think about how ROI will change in 3-5 years. For example, is shift-left testing worth using in your practice? Shift-left testing is a methodology that focuses on performing testing as soon as possible, starting with the requirements collection phase. The idea is to think about the errors in advance and find them as early as possible, because it is believed that the error found at a later stage in the life cycle of the system development will be more expensive than the error that was found at the initial stages.

Have you appreciated the capabilities of your organization?

You need to match the capabilities of your organization with a stack of test automation tools. Successful implementation of an automated testing strategy requires both product knowledge and automation knowledge. Your team should have a clear understanding of how to use the intended automation tool, as well as how the application works.

Test support is important

Test support is another factor that people tend to overlook when measuring ROI from test automation. If you use Selenium for automatic cross-browser testing, then after the successful implementation of your strategy, you will need to regularly update and maintain test cases. The regression set and test cases will begin to grow in number as you add new pages, expand or update the functions of the web application. Keeping these test suites usable for a longer period of time will require regular support.

Lack of proper quality documentation

This is a fairly common mistake, and not only in terms of automation, but also in terms of management. The documentation should reflect the organization’s standards. When the tester writes a test script, he must prepare a document explaining the purpose of the script and how it works. A common knowledge base should be accessible to everyone within the organization and include documentation for each automation scenario used in your organization. Such a knowledge base will serve as a reliable support for any newcomer to your team. It will also help eliminate problems that inevitably arise in the absence of a “senior” project manager or when a valuable tester leaves another company. Therefore, the next time you calculate ROI from test automation, be sure to take into account the amount of effort required to maintain one such knowledge base for internal employees.

What to do to increase ROI from test automation with Selenium

At the moment, we are aware of common errors and know the indicators for calculating the ROI from test automation. What next? Performance. How to find the best way to get the maximum ROI from test automation with Selenium? Well, here are some noteworthy practices to help you get the most out of your test automation.

Implement Automation for New Tests

This is a very important factor, especially if you are moving from manual testing to automated testing. Suppose you introduce the use of Selenium WebDriver in your organization for automatic cross-browser testing.

• Calculate the number of test cases that need to be automated;
  - Now check which of them really need to be automated and which ones are best done manually;
  - Do not change test cases in such a way that they can be automated! There are some things that need to be checked manually;
• Calculate the hourly cost of a tester performing test cases;
• If some testers do not have automation experience, calculate the cost of training.

The priority of the process of automation of new test cases

We all know that regression testing is always a priority, especially when it comes to visual regression testing of a web application in various browsers to check cross-browser compatibility .

Regression testing mainly involves re-executing old test cases to make sure that the new functions that were added did not affect the operation of the old functionality and did not cause new errors. Over time, as your web applications grow architecturally and functionally, manually tracking the progress of regression testing will prove costly. Implementation of automated visual regression testing makes sense if you want to save.

• When calculating ROI, assume that each new test case will soon become part of regression testing. Treat them as part of your regression testing strategy.
• Sort test cases by complexity and see which ones can be automated.
• As stated earlier, consider the cost of maintaining old tests.

Almost 100% test coverage under various test configurations for browsers and OS

The main goal of test automation is to improve the quality of your application. When calculating the ROI, you should also take into account that the number of ways to view a website is increasing every day. There are hundreds of browsers and devices on the market where people can view your web application, and their number is constantly growing. Define a browser compatibility matrix for yourself.

Coverage Ratio Recommendations

You can increase test coverage by performing smoke tests, unit tests, regression tests that point to the source of the errors.

Unit tests. Unit tests make up the largest percentage of tests when testing your web application. It always makes sense to use a parallel execution mechanism to save time.

Smoke tests Running smoke tests in parallel is the best way to cover test cases when sending hotfixes to an application. Automated smoke tests are a good way to test daily for your web application.

Regression tests. In the Agile age, rapid development requires more and more regression tests, especially for version control. Performing parallel regression tests ensures that each subsequent assembly will work exactly the same as the previous one, which will make it possible to fit the tests in a significantly shorter period of time.

Remember about defect leakage - the number of errors that occur during the production cycle, due to the fact that they were not detected at earlier stages of testing. This can happen due to poor functional test coverage or poor testing environment.

Try using the shift-left test methodology. It implies the involvement of testers in development. Developers also participate in it, performing unit tests after the development of a specific module. The main idea is to find the error before it shows itself, and ultimately reduce its cost.

Find out the number of duplicate and redundant test cases

Duplication of test cases is an important factor that can lead to an increase in the budget for testing. There is no point in developing again those test cases that you already used earlier for another module. Reuse of test cases leads to faster testing.

Cost calculation includes:

• number of repeating tests;
• test cases with repeating components;
• the time required to detect and create all these redundant test cases;
• calculation of the cost of using the test case management tool.

Recommendations for Reducing Redundancy:

• Use the test case management tool to find duplicate scenarios. You can use these tools to store tests with variable fields, which can later be modified to your needs. Using such a tool will help to quickly detect redundancy.
• You can also develop modular test cases that you can reuse in the future. Highlight the tests that run most often. For example, login and logout functionality. To check the correct operation of these functions, you need to conduct a certain range of tests. Design a unit test case that can be used for each of the many variations of logging in and logging out.

Perform automated cross-browser testing with Selenium Grid On-Cloud without problems

The imperative area of ​​ROI calculation from test automation with Selenium lies around the execution method. Selenium is known to be an open testing automation framework designed to facilitate testing of web applications. You can conduct automated testing with Selenium either locally or using one of the cloud tools offered by Selenium Grid.

When you do automated testing with Selenium in your own infrastructure, you need to consider the budget to scale your test suite. How will you introduce new devices? Newer browser versions? To support parallel execution, your computer will require significant power. However, if you are testing with the Selenium Grid in the cloud , you can scale it to fit your project requirements.

Selenium alone does not provide a reporting tool. Test reports can be obtained using test automation frameworks based on the language used. If you use LambdaTest's Selenium Grid, you can get reports through our open Selenium API .

Another key difference between the two methods is parallel testing. Using Selenium Grid on your local machine, you can run test cases only in browsers installed on this machine. However, if you use a cloud-based Selenium Grid, such as LambdaTest, you can test on more than 2000 real browsers and their versions.

LambdaTest is a cloud for cross-browser testing in which you can run several Selenium test cases on several combinations of devices and browsers in parallel, which reduces the cost of creating your own “laboratory” for cross-browser testing and testing on various devices.

Remember that if parallel testing is not performed correctly, there is a risk of defect leakage. The sooner you find a mistake, the easier it is to fix it.

Advanced ROI Calculation Methods

Now that we’ve learned the basics, it's time to talk about the advanced methods used to calculate ROI.

Effective ROI

a source

Because automated test cases can be run all day, ROI is calculated in days. On the other hand, with manual testing, only the tester’s work time is taken into account, which averages 8 hours a day. The formula that is the basis for calculating ROI:

• (a) Time to develop a test case for automation = (Time to automate one test in hours * Number of test cases for automation * ROI period) / 8
• (b) Automated test case lead time = (Automated test lead time per test * Number of test cases for automation * ROI period) / 18
• ( c ) Automated test analysis time = (Test analysis time * ROI period) / 8
• (d) Service time for automated test = (Service time * ROI period) / 8
• (e) Manual execution time = (Manual test execution time * Number of manual test cases * ROI period) / 8

Note: The payback period of investments (ROI Period) is the number of weeks for which ROI divided by 8 should be calculated, occurs everywhere where manual labor is taken into account. Division by 18 occurs wherever automation is used.

In calculating effectiveness, the focus is on how effective automated testing is for the organization. The monetary factor is considered secondary, therefore, it is not necessary to include hourly pay for the work of the tester.

ROI to reduce risk

It is supposed to independently calculate the benefits of automation. We will again take an example with cross-browser testing and using WebDriver to understand how this works. During manual testing, the entire team spent a lot of time doing the same things over and over in multiple browsers. After the introduction of automation, testers have a lot of additional time to perform productive work, such as developing test cases, analyzing an application, etc. In short, ROI to reduce risk solves problems that have not even been raised before.

With the introduction of automation, test coverage is increasing. Full dependence on manual testing will lead to unnecessary errors that may appear already on production. This will lead to a decrease in product quality and testing efficiency. Such a possible loss is considered a risk. There are no changes in the value of the investment. Only monetary losses that an organization may encounter without implementing automation are calculated.

This is just a wrapper! (orig. It's A Wrap!)

Thank you for reading, I hope that you have gained the necessary knowledge and a useful strategy for convincing the management with the help of ROI from testing automation with Selenium. Make no mistake thinking that ROI is just a comparison of time and money spent manually testing a web application and automated testing. There are many factors to consider, and each of them has its advantages and disadvantages. In addition, ROI cannot be calculated with a single formula. There are several methods, and you will need to choose what you need depending on the requirements of your organization and your project. Good luck in testing!