Exposing antivirus tests

This week, Chad Skipper from Cylance has published an article entitled Security Testing Houses: Know the Truth! Which should be read by people interested in testing security solutions. It contains a number of serious accusations against some test laboratories and manufacturers (without specifying their names), for example:
- manufacturers pay to test the results of their products showed 100% efficiency
- bribing test laboratories to hide negative test results.

Although I have been working in this industry for over 17 years, I don’t know a single case described above. But at the same time I agree with most of the article. I will name only a few points:
')
1. Outdated testing methods;
2. Lack of used samples;
3. The need to pay for participation in testing, etc.

And it needs to be fixed. That is why there are organizations such as AMTSO (Organization for the Development of Standards for Testing Antimalware Solutions), and the first thing that comes to my mind after reading the above article is that "we need to invite Chad to the next AMTSO event." But it turns out, when I said this at AMTSO, I was informed that he had already registered for the nearest event, which will be held next month in Malaga. Wonderful!

Chad ended his article with the words "Test for Yourself." I also agree with this, and in fact, this is exactly what has been happening for a long time. Our very large customers from various sectors of the economy (government organizations, telecommunications, finance, healthcare, industry) chose our EDR solution ( Adaptive Defense 360 ) after having tested various solutions intensively and thoroughly for several months.

The truth is that this kind of do-it-yourself do-it-yourself tests are available only for large corporations . Small and medium enterprises do not have enough resources to carry out thorough testing correctly, and therefore, when making their own decisions, they trust the results of professional test laboratories.

Kevin Townsend from Security Week wrote a few months ago about this issue in his fantastic article “Inside the Competitive Testing Battlefield of Endpoint Security” .

Among all the regular tests conducted by the largest test laboratories, one of my most favorite tests is the Real-World Protection Test, performed by the AV-Comparatives laboratory. In the generalized test report for February-June 2016 with 1868 tests ( PDF ), check whether many manufacturers received 100% results from 0 false positives? No one. Obviously, Chad cannot imply AV-Comparatives when it says that manufacturers pay money for getting results with 100% efficiency.

I said the same thing in AV-Comparatives last year, offering to test our EDR-solution Adaptive Defense 360 ​​in comparison with other similar solutions. Have you seen this test? No, because even if Panda offered to pay for each product included in this test, other manufacturers (Cylance is NOT one of them) do not want.

After 3 weeks in Denver, I plan to discuss these issues at the 26th Virus Bulletin conference with Richard Zvinenberg from ESET in our discussion section on Anti-malware Testing Undercover .

Article author: Louis Corrons