How to test antivirus software

For the last five years, I have been engaged in helping to make testing system applications faster and cheaper. In my youth (being young and hot) I tried to just work faster with my hands and more attentively with my head, but gradually I realized that manual testing has a limit of efficiency: for a department from a manager and two testers, this is about 30 features per month. It would seem that there are no problems, because there is a well-established path for ruthless test automation - one getting rid of regress will give you more than enough free time, we will simply choose a system from what is on the market, we will implement and we will live happily. And then life declares: "No."
And the point is not that all autotest systems have a fatal flaw , or do not meet a simple list of requirements generated by my wild imagination ( link to my requirements ), or cannot display all the unimaginable complexity of user use cases - a simple analysis tells us that the user is using -cases of any antivirus must include relatively simple steps:

1. Deploy to a clean machine
2. Checking the static characteristics of antivirus files
3. Treatment and detection of various threats
4. Uninstall
5. Reinstall
6. Old version update
7. Compatibility of the installation with competitors' antivirus solutions
8. Self defense tests

I would particularly like to note the point about compatibility - it seemed normal for me that the antivirus is someone from the list of kaspersky, drweb, eset, symantec, etc, but the world did not stand still and the users of the new generation did not find anything for themselves - now Parse the download.cnet.com/s/antivirus page with popular downloads for the week and be tested with what is relevant.
As you can see, all cases directly ask for automation.
But in testing system software, and antiviruses including, there are a couple of problems:

• Work on hardware is different from work in vm, among the test machines can and should be both physical and virtual environments.
• The second problem of testing system software was noted in 2013 by the talented researcher Fridland Yu. M. and it sounds like “providing support for reloading in the system of autotests”.

Almost a month of formations of various queries to search engines led me to the fact that there are no other relevant answers to the question “how to do auto-tests with reboot” except for a presentation to the graduation project of our partners . Oh gods, I thought, after reading that pdf - dragging a whole TFS with all the associated Microsoft infrastructure for life after a reboot?

Why do I need it?
And I began to study other solutions. Yes, yes, I did watch docker, read jenkins-based ci, tried and disappointed - either the solution did not fit completely, or the number of necessary related technologies gave a uniquely modified unsupported solution to the study (or supported by a team of highly qualified specialists), I I changed projects and employers (for more and more famous ones, of course), but I saw the same thing everywhere - testing the system functionality (driver update, OS shutdown, virus treatment, autoload check) the fastest, most reliable and cheapest way.
And then I discovered python. On this technology alone, it was possible to write a simple virtual machine manager (import pysphere), a web interface to it (import web2py), a testing agent — a win32 service (import win32service) delivered to the virtual machine through an iso image, and the tests themselves compiled into separate exe-files via pyinstaller (so as not to suffer with the preliminary preparation of systems, the installation of python etc). Conceptually, it all looks like this:

All this stuff is supported by 1 specialist with knowledge of 1 technology, the bus factor is minimal, and I am extremely happy about it.

The results of the tests are collected in the day report page:

In this simple way, we got rid of regression and smoke tests in about half a year of development of our bike, only new functionality falls on manual tests, from which the entire testing department smoothed out wrinkles and hair became soft and silky, which is what you want.


Actually, after recruiting a number of new features, the product is given to the probe team - beta testers, and through technical support in technical support, the quality is brought to unprecedented heights.