The introduction of automatic testing

When using automatic tests for quality control of numerous assemblies of our software solutions, we, no doubt, received the initial gain compared to manual testing. At the same time, however, the convention, or incompleteness of automation, associated with a large amount of manual operations to start, adjust and process results, significantly reduced the efficiency of the process, forming an overall semi-automatic hybrid in testing.
What kind of problems took our time and effort, as well as in any other company operating at the same level of “semi-automatics”?

Three types of scripts

For automatic testing, at least three types of scripts were used. Of course, this was due to the specifics of the tests, but this was no easier for us. Each script required its own special launch procedure, which was performed manually between launches of the next automatic test.

Setting the sequence

Setting the test execution sequence, also performed manually, took considerable time and required considerable effort.

Setting up the environment

Each automatic test required its own original environment settings - moving necessary files from different places, requests, etc. All these operations, performed again by hand, gave a very clear idea of ​​the first part in the definition of “semi-automatics”.
')

Receiving reports

Each type of script gave a report in its form; two of them were completely unreadable for the manager, allowing you to get the right information only at the cost of tremendous effort and effort, not to mention the effort of switching from one form to another.

Faced with the above problems, I began to look for ways to solve these. We agreed with the boss that you could write a complex for managing automatic tests (the developers had free time at that time). And that's what we got after 3 months of development (1 developer + 1 tester, for a total of 6 person-months).

Autonomous testing process

The automated test management system, which covers the entire software development cycle — from assembling the next version of the product to the final generation of the test report — is based on Visual Studio 2008 & Hyper-V. The created system provides the possibility of transition from the “Developer - Tester - Developer” scheme to the actually existing “Developer - Automated System - Developer” scheme with a sharp improvement in the quality and level of testing. Due to its versatility, functional power and flexibility it allows you to perform any prepared tests on any software products.

Any tests on any products

The system, as a single tool for performing a huge number of a wide variety of tests (Visual Studio MS tests, HP QTP, etc.), while covering the entire software development cycle, allows you to automatically capture detected defects on TFS, managing each stage of the cycle; expand and parameterize the automatic testing system itself, based on the amount of available resources, the number, type and purpose of tests and test products.
Friendly user interface makes it easy to perform many operations to manage and monitor the actual status of expected / executed / executed tests, test planning on different versions of the product with setting priorities, parameterizing the system depending on the environment and operating conditions, and much more.
A logging system that uses various technologies flexibly for different stages of testing allows to obtain accurate and detailed information on the execution of a test sequence and the problems encountered.
The automatic registration of defects on TFS without the participation of a tester, which includes both opening and closing records with the possibility of parametrizing this function, releases a considerable reserve of temporary company resources.
The extensible data architecture of the system allows the use of modern technologies for storing test information on different resources - a file server, on Source Control Visual Studio 2008, on iSCSI.
These are only the main features of the automatic test management system; being in the process of continuous development and improvement, this unique software product is steadily acquiring new functionality and special reliability, making it an indispensable tool for any company that produces or tests software.

At the moment we are working on creating videos that show how the system works in real conditions. Once done, post on Yuotub.
If someone is interested in our system - write, I will always be happy to answer your questions.

UPD : at the request of workers united in one topic. Soon add dry technical data. Let it be for now:
Octopus works with TFS 2008 as a build machine + BTS, HyperV as a virtualization system.