Automated regression testing approach

Hello dear readers. We would like to coincide today's material with the launch of the "Python QA Engineer" course. Anticipating possible questions, we warn that the article does not contain a word about Python, but we still find this material useful for testers, so we decided to share it with you.

Testing every smallest detail of the code is an impossible thing, so regression testing should perform a comprehensive check and focus on a specific area in its entirety. The main goal is to ensure that no regression error will damage the critical business process. It is this effort that allows you to benefit from automation. An automated testing approach that focuses on reducing regression errors will help go a long way toward building good customer relationships and increasing brand value.

My team is responsible for testing the application for accounting, which uses complex calculations and puts the ledger records into the accounting system. All of this is a workflow, and closing books every month is the most important.

With each release, there are some changes in the calculations, for example, an account may need to increase debit or credit, or the available value may be divided between two accounts. Such changes in a complex application that has many internal and external dependencies often lead to unpredictable behavior, including regression errors, in unexpected and seemingly unrelated places.

Let's deal with the regression error, sometimes referred to as a critical incident or production problem. Having worked for several years in the software industry, you understand that the regression error leaked to production is much more important than a wrongly functioning new function. The fact is that the new function is still hidden, so when it breaks down, the impact on the business component is minimal, whereas with a regression error, the user relies on the work function and most likely does not have a backup version of the work system.

Regression errors are caused by changes that were not taken into account by the project manager or the product owner during the acceptance tests; architect, developer during code review at the design or implementation stage; or QA-analyst and tester in test cases. All security networks missed the error and a user came across it. If an error in a recent update was detected at any of the above stages, i.e. teams, interested parties or any other link that is present in your organization, then it was reviewed prior to release, or at least assessed for criticality.

Regression errors entail several costs: urgent correction, fines for potential violation of the service level agreement and, most importantly, they damage your users' trust in your product. They may decide that a new release will break something that is already working.

It became important for my team to check everything.
However, it is well known that this is impossible, or at least too expensive and time consuming. Therefore, “testing everything” focused on the following two things: critical business processes and confidence that the behavior in the main critical business processes will be the same as in the last release before the change. In general, we wanted to make sure that a regression error did not appear in the latest release in the critical business process.

We evaluated our typical automated testing approaches to validate calculations and verify that all of the logic is described in code to automate testing. This approach led to the classic question: “Who is testing the code of the tester?” If the test case was unsuccessful, the same probability remained that the problem was in the code of the tester. In addition, with each change in the application, the test code also needs to be updated, and such changes happened often.

Also, thanks to automated testing, we made sure that we have a fixed set of inputs and an already known set of outputs. Due to the complexity of the application, it was impossible to submit all possible input data sets at once. Also needed were various data sets for the end of the month, quarter, year. The schedule was a serious problem, since it takes a long time to generate a huge set of input data and the corresponding scripts.

There was another variable: user data. We had a special privilege: we received backup copies of user data every month. The quality of the test is directly dependent on the data used for testing, and the production data is always better than the generated data, so our privilege was a huge advantage that we did not want to lose.

Identify and implement regression tests

Our idea was to use automated testing, which needs the minimum necessary maintenance, and which strengthens the confidence of stakeholders that the release will be of good quality.

So, we needed a testing strategy for critical business cases that would guarantee the absence of regression errors and, of course, that we could quickly put into practice.
Our preparation process looked like this:

• Backup database with production, restored twice;
• Two test systems are installed, which work in parallel:
  
  • One with a production code;
  • Another one with the current version of the application under test.

This approach provides two identical systems with differences in the code in just one version:
Having two systems is quite important because it helps to understand that any problem occurs only due to the latest code changes.
Tests are separated, so from the standard process “perform an action and get a reaction” we move to the fact that actions are performed from one point to another while preserving the workflow, after which the error reports are compared. This is the key to identifying unexpected errors.

When the tester focuses on a new function or some change, the test turns out to be oriented specifically to it, i.e. verifies the relevance of a specific change. Regression testing is different in that it must verify that nothing has changed. This difference is reflected in automation scripts. It makes scripts testing specific functions unsuitable for finding regression errors, so a different approach is needed here.

For example, if you are working with an order management system, you need a script to place orders with a lot of input data to place orders for two installed test systems (preferably working in parallel), then you get a report on the orders for the past day and compare each value. Then all orders are confirmed or approved (this action), and reports, such as daily confirmed orders, orders by nomenclature, reports from the warehouse, orders from each carrier, type of shipment, type of payment, etc. will be compared in two systems. This continues throughout the entire workflow. You can combine actions, such as placing and confirming orders, and then compare reports at separate stages.

Another example is a hotel management system, in which individual actions are defined as critical business processes, such as check-in, billing in a restaurant, and receiving inventory. All these processes will be assigned their own actions and reports. The difference in this system compared to the previous example is that test suites can be run in parallel, and there is no need to complete a step before starting the next one.

A comparison of two reports is a moment of truth, and it must be flawless, in the sense that no stakeholder should have doubts about its correctness. The difference in the reports is the real difference.

For this operation, we use the web service interface. All report submissions are performed in parallel on two systems, and the resulting response is compared in JSON format.

The comparison takes place on three fronts:

• Source (production) minus target (application under test);
• Target minus source;
• Comparing values ​​to get the intersection.

This will work for XML, XLS, fixed-width CSV or any other format. We need to make sure that there are no unnecessary records, no missing records, and all the values ​​of the records are the same.

This is the essence of the approach we are talking about. All this is information available for reading in an application that is issued as a report or, in some cases, acts as an interface to other applications.

The success of this approach lies in a comparison tool or utility that processes cases related to your application. You can consider yourself lucky if you find something suitable out of the box, otherwise, it is important to understand that investing in such a tool is worth it, because they will bring good results.

After all the talk about automation, you need to insert a remark. Since some differences in the reports are expected, as they must be there in accordance with the requirements, all results must also be analyzed manually. There should be clear successful results of passing test cases, however, unsuccessful results should also be analyzed and their validity must be confirmed. Since we are talking about regression testing errors, they should be corrected before release. Of course, there are some exceptions that are handled in accordance with the application.

Program setting

All applications are different, and installing and configuring them also happens in different ways. Some additional steps may be required to prepare an application for testing, so they need to be considered at the right time and in the right place to perform the tests. Here is a set of typical steps:

• “Confuse” the data from the production by deleting e-mail identifiers or other confidential information, or replace it with dummy data;

  
  

• Receive data in proper form to run the test;

  
  

• Adapt settings for QA-environment, for example, by changing the integration of communication.

  
  

  The only point to remember is that the listed actions must be performed for both installations. Remember that before you start executing a test suite, the settings must be identical.

  
  

  Often, actions other than a report request may return an object, for example, an action, such as creating or modifying an order, may return a new order object. In this case, you need to compare two objects and not wait for the comparison of reports. This can help identify the error in the early stages at the root.

  
  

  It is also recommended to break the entire set into smaller sets, for example, by grouping transactions and related reports together. Kits can be run in parallel to save execution time. However, for an application with a characteristic workflow, this will only work if you can split the cases vertically and horizontally or vice versa.

  
  

  Variations can begin with technologies - JSON, XML or scalers (int / string / float), and expand to the point that the application under test and production will respond with different structures, but still correspond to the architecture. For example, the production version can use the old JAR file that operates on a specific format, and in the new version the JAR file has been updated and now the response format has changed, so comparing them will show inconsistencies. In order to compare them properly, you will need a temporary plugin.

  
  

  There will probably be few such situations. In such cases, it is sometimes easier to tweak the design or view them in the context of a workaround.

  
  

  There are several options for processing such comparisons:

  
  

• Ignore some fields, such as identifiers and dates;

  
  

• Ignore numeric differences less than 0.0001;

  
  

• Ignore case sensitivity;

  
  

• Structure changes in two responses.

  
  

Improve regression testing

Regression testing should be holistic and focus on the whole area. This balance will help to benefit from automation. An automated testing approach will help reduce the number of regression errors and help on the path to good customer relationships and increasing brand value.

Now, in the rhythm of constant movement forward, our team wants to try to abandon two identical system settings that we are using now and implement the same strategy on one installation. We want to save responses from past executions and use them for comparison. The hike to testing can always be improved. Wish us good luck with this!

Translation prepared specifically for students of the course "Python QA Engineer" .