Clean and dirty test environments

Good time of day, Habr! In this post I would like to talk about test environments, what is a clean and dirty environment, what tests are performed in them and what tasks are achieved. If you are not indifferent to these issues, welcome under cat.

Each software developer, ever tested his offspring, sometimes without even knowing it. Whether it is an application for windows, and maybe a mobile application for Android or iOS ... As a rule, novice developers are limited to test runs in the environment in which development takes place. With the professional growth of a developer, the complexity of his creations grows. Simple test runs can not do. Then manual testing of the functional is used. With a further increase in the complexity and age of the product, the number of functions that need to be covered with testing increases, at the same time it is necessary to remember the old functions, the performance of which should be checked. The latter are not efficient to process manually and automatic testing takes place on the scene.

Of course, it is not efficient to perform automated testing only in the development environment, since it is limited to one platform, one operating system, one set of installed software and traces of the presence of a product left over from past test runs. The problem of testing the functionality of the platform, the operating system is solved by increasing the number of test environments: for example, increasing virtual or physical machines with different operating systems on which automatic tests will be run. The more operating systems covered, the better the product will be. All operating systems must be up to date.

With physical machines, everything is simple: just enable auto-update. With virtual machines deployed from images, the situation is more complicated. To keep them up to date, you need to update the image. The frequency depends on the moment of the release of the service packs or product-critical updates of the operating system. In addition, in itself, it is useful to update the image. Our small test automation group Acronis TrueImage made an willful decision: this period equated to three months.
')
As in any business, you shouldn’t go to extremes, all variations of operating system versions are very difficult and inefficient to take into account. It is highly likely that the user will have the current version of the Operating System (OS). And it is worth being guided by it.

What is a clean test environment?

Problems of traces of the presence of previous versions of the product and interaction with third-party programs are solved by adding environments to the test plan that are as close as possible to the user's real environment (the so-called dirty environment ) and minimalistic environment (the so-called pure environment ). Both species have the right to life and their purpose in the testing process.



A clean environment - ideally, this is a newly installed operating system, followed by installing a product on it. The number of external factors affecting the performance of the product is minimal. Test "cooked in its own juice." There are no traces of past product launches, any interaction with third-party programs, for example, antivirus or firewall, is excluded. Thus, in the case of violations in the functional, we can confidently say that the problem is in the product. Even if the problem is in the OS, the developer must ensure that it is working, since the OS is at a lower level. To the flaws of the OS must adapt or reconcile. OS rules the ball. As a result, a clean environment facilitates the localization of bugs, reduces the time for setting up and deploying the test environment.

As a pure test environment in which TrueImage is brewed, there is a small flock of physical machines with a population of 5-10 individuals. They are usually at the disposal of manual testing engineers.


Shot from the movie where the valiant testers engineers localize bugs

In our arsenal, there is still a horde of virtual machines, most of which live on blade servers. Virtual machines are considered conditionally clean environments, as they are defiled by our python scripts, which are copied to the experimental virtual machine in order to verify the functionality of the product. The lifetime of virtual machines is short and equals the lifetime of the test script. After running the virtual machine, if there are no bugs with honors in the form of a “green” log, it is deleted from the server. But if there is a hint of a defect in the application being tested, the virtual machine, after folding the logs into a secluded storage, is frozen for further clarification of the circumstances.



And then our magic wand comes in, which saves a lot of time - a program for analyzing logs for known bugs. The principle of its operation is not complicated, since logies are written by a computer, and not by an unpredictable person, it is enough to find the corresponding phrases or sets of phrases characteristic of the bug. By analogy with the antivirus, a database of characteristic phrases and manifestations of defects is maintained. Unfortunately, we have not yet written our antibagus with blackjack and heuristic analysis, so that if an unknown defect is encountered, we have to personally go into the logs.

... and with which this clean environment is eaten

We have decided to divide the tests at several levels depending on the coverage of the functional and the execution time. The first level is the BVT level (P1) - tests covering the main critical functionality, without which you shouldn’t even run the product: system backup, subsequent restor, etc. The next level is called Common (P2-P3). It includes functional checks, without which you can use the product somehow: backup disks of different file systems, restore to disks, with a size different from that stored, a scheduler. BVT and Common it is advisable to perform, first of all, in clean environments. Since these test suites are performed at the stage of active product development, it is important to run them regularly and, if bugs are found, they can be easily localized.

And what is the dirty test environment ...

With clean environments where only the OS is present and the product is sorted out. But the software world is not only a clean operating system. And in the world, in addition to the product being tested, there are other programs, even other versions of the product itself. And how they affect the functionality of the test subject would also be good to check. This is where the so-called dirty environment comes into play.


Dirty is not always bad ...

Dirty environment - an environment in varying degrees close to the real environment in which the product will be used: the composition of the installed drivers, additional software, hardware components. It may also be influenced by the presence of traces from previous launches of previous versions of the product, the use of accounts (for example, in cloud services) containing secondary data (backups in the cloud).

Since different environments have different roles, respectively, the test suites for them are also different.

Why do we need to muddy?

Now it is the turn to tell what checks are performed in a dirty environment.

What product function should you check? Operation restaura? Wasn’t it all being done for her? Is it not after this operation that the stone falls from the shoulders? Not. The user can live in peace and well, knowing that he has a backup under the pillow. So, backup should definitely pay attention. But there is another operation, without which there will be no backup or restaurant. This is a product installation operation.

Acronis pays a lot of attention to this operation. There is a type of test covering the entire life cycle of a product - LifeCycle. These tests consist of steps such as: install, uninstall, reinstall, update, upgrade, etc. Plus, the chains take into account the rich history of our products: 10 major versions - upgrades (but only the last 5 versions participate in testing, including new versions of the product: TrueImage 2013, TrueImage 2014) each of which has from 2 to 5 minor versions - grocery updates. And this whole zoo is required to check. Will TrueImage 2014 update2 be installed if the computer has already installed TrueImage 2013 update1 version? And upgrade TrueImage 2013 RTM to TrueImage 2014 update1? These questions will be answered by the above LifeCycle tests. But this was not enough. It is not enough just to put the product. It would be nice to make sure that it works. Therefore, between installations, upgrades, updates and reinstallations, the basic functionality of BVT is checked.


Talisman TrueImage. To magnify homy. In honor of the former name TrueImage Home

Another type of tests performed in a dirty environment, the so-called Longhaul or Stability tests. They are performed to assess the stability of the product over time. Their essence is in performing the basic operations of the BVT, concluded in an automatic cycle for a long period of time. We have this segment is 7-14 days, then the build is updated and the cycle of tests is repeated. Thus, the previous cycle of tests affects the next one. The backup and restore of large amounts of information sometimes take a single day, especially from the cloud, and especially with a slow channel. Also, such tests reveal bugs associated with long work, for example, a memory leak, filling the hard disk, etc.

Peace, friendship, chewing gum

But that's not all. But what about the influence of third-party programs on the product? Here, unfortunately, automatic testing is not effective. The computer can not fully assess the correctness of the interactions of the product and third-party programs.



The most serious problems arise with programs that control access: antivirus and firewalls. Here come to the aid of manual testing. For example, will the standard Windows firewall allow a previously prepared backup to reside in the cloud?

Tests are different, tests are important.

Splitting tests into different levels makes it possible to choose trade-off solutions: the time needed to complete tests is inversely dependent on coverage. For example, BVT tests are performed in the shortest period of time and allow you to quickly identify critical and blocking bugs. Common already takes much more time, but it also covers more functionality. Such tests, run regularly and frequently, allowing “to keep abreast”, as a rule, are launched in clean environments .
LifeCycle and tests on the impact of third-party software makes sense to run at least not earlier than the beta testing stage, when the product is stable. Of particular importance is stability tests. They are performed on individual machines 24 hours a day. In this case, we already need a dirty environment .

The presence of different levels of tests and different test environments provides greater opportunities for the localization of bugs. The lower the test level, the less time it takes and the more often it is performed. Suppose a bug was detected when performing a test for interaction with third-party software (high level). This bug also manifested itself when performing Common tests, but at the same time was not seen at the lowest BVT level, it can be concluded that third-party software had nothing to do with it, since the bug was found in the pure Common Test environment, but not in BVT tests. So it is in the functionality covered by Common tests.


A clean environment helps to quickly identify and localize bugs at low cost. The dirty environment on the contrary helps to find the most sophisticated defects in the product. Automation justifies the funds spent on its implementation, but it will not completely save you from manual testing, at the same time some things, such as the product life cycle, cannot be tested without automation.