Test can not be pardoned
Hi, Habrovchane! The fact is that Evgeny Kaspersky posted another witty text on his blog, and we think that you should read it too! What do you think about it?
As expected, the recent post about the Passmark performance test made some noise. And the main thing - the noise was not so much in the blog, as in the antimalvarnoy industry. Well, this is exactly what the real Montmorency usually wants :)
In short - it turned out. Because I want to find the truth. And not only that - I would very much like to somehow stimulate the correctness of antivirus testing.
')
And how do you need to test antiviruses?
Yeah, so firstly. In order to avoid raids. To immediately bring down komenty, that type we will now teach testers to cook stool according to our recipes, so that our products climbed into the lead. There is an answer to this - we rarely do NOT fall into the top three in various tests anyway. And secondly, the fact that what is being discussed now is not my invention, but the AMTSO industrial standards (Anti-Malware Testing Standards Organization is such an organization), in which there are representatives of almost all leading AV-vendors and various well-known experts. .
Alas, the general public and the progressive world community know little about AMTSO, and most testers still prefer to do their work in the old manner. It is understandable - it is cheap and familiar, and the user, they say, the main thing is the result - who took the first, second, third place, and who went to the dump.
It seems everyone is happy, BUT! It really distorts the picture. In the end, it turns out that not the best software creeps upward, the position in the list of test winners practically does not correspond to the real level of protection provided - in short, bullshit and uh ... consumer fraud, in short.
Why am I so hot — sometimes it is just a shame that time and resources are spent not to “do business” —but to provide demonstrations in the next shit testing. To show the result is not worse than those that stupidly sharpens their products not on real quality - but only to test more correctly the rest.
Here it is. And now - let's get to the main point.
So, how not to test.
Classic, good old- on-demand test .
Actually, this is the most common standard and familiar test, and once, long ago, in pre-massively Internet times, it really was the most correct one.
// we, by the way, in 1994 staged the international premiere of AVP just on such a test of the Hamburg University - we participated for the first time and immediately broke everyone.
The testing technique is as follows: the disk is taken and scored by Malvara, the more - the better, the most varied, to which the arms have reached. Then various anti-virus scanners are set on him and the number of detectors is measured. Cheap and angry. But already 10 years old as absolutely irrelevant !
Why? And because anti-virus signature, heuristic and other “scanning” engines are only part of a set of technologies that are used in the real world for protection! (and the value of these engines in the general level of protection is rapidly falling). Often, the scanner generally works as a last resort for purely surgical work: for example, System Watcher tracks some Trojan, understands the infection pattern, and then transfers the task of ejection to the scanner.
Another drawback is the base for Malvari to scan.
There are two extremes and both vicious. Too few malware files - you know, irrelevant. Too many samples - the same trabla, but from the other end: too much garbage gets into mega-collections (broken files, data files, not-so-good malware, for example, scripts that use malware, etc.) and clean collection from such garbage - the hardest and ungrateful work. In addition, low paid :)
And the main drawback - under such tests, you can sharpen any product. To show outstanding results in these tests. Fitting the product to a test is done elementary - you just need to detect the files that are used in the test to the maximum. Do you follow the train of thought?
Again.
In order to show in the "scan tests" close to 100% result, do not rest against and improve the quality of technology. You just have to detect everything that gets into these tests. Something like a joke about hunters and bears:
- But the bear runs faster than us! - says the first hunter.
“And I don’t need to run faster than a bear,” answers the second, “I just need to run faster than you.”
To win these tests do not need to "run faster than a bear." You just need to cling to the sources of Malvari, which use the most famous testers (and these sources are known - VirusTotal, Jotti and Malvaro-exchangers of anti-virus companies) - and stupidly detect everything that others detect. Those. if the file is detected by competitors - just detect it on MD5 or something like that.
Everything! I am personally ready from scratch , by a couple of developers , to make a scanner that will show almost 100% of detections in a couple of months. // why exactly two developers are needed - just in case, suddenly one gets sick.
In short, on-demand tests are incorrect tests. Since they do not show anything real, it is easy to adjust to them and it is extremely difficult to win honestly.
How can you test, but where NOT SPECIAL is better not to look.
There is a zoo of all kinds of niche tests that show the quality of antiviruses in some very specific area .
In principle, they have the right to exist; moreover, they are extremely useful for comparing the quality of production of any particular feature. But it is necessary to write in BIG letters about this specifics and that the test does not take into account all the abilities of the product . What the average user has nothing to do here is purely industrial tests that carry a payload only for specialists and other geeks.
For example, a test for treating a specific case (how a product copes with treating a system that is infected in a certain way), a test for false positives (like the product "Falsit" on pure software), a proactive test (how the product catches malware without signatures, i.e. proactive technologies), on-access test (quality of on-access scanner for real-time operations with malware), measurements of interface performance and gagging , etc. - various tests for a specific purpose.
The type of individual tests for acceleration, deceleration, and gasoline consumption - but only separately, without reference to each other. Since the best result can show an absolutely non-usable product in everyday life, such as a Formula 1 car :)
Finally, how to test and where to look ...
I'll start from afar. What are tests for?
First of all, to show the quality of protection. Yes, with all equal usability and performance are also important, but, ultimately, we are “to go, not checkers.” Therefore so.
How to check the quality of protection?
Of course, in an environment as close as possible to combat ! The methodology of the correct test should be based on the most common scenarios for users in real life . All the rest is a compromise and formalities from the category “if it were yes, if it were so.”
For this task, the dynamic or real world test is best suited.
The meaning is simple: we take an ordinary computer, put an antivirus on it with the default settings and try to launch the current malware in every possible way. It's simple!
The product works at full capacity, the situation is as close as possible to the combat! And the user as a result receives the correct measurement of quality of an antivirus and the relevant information for a reasonable choice. Then measure the load on the system, the size of updates, add the price, put “weight per points” for each category of testing - and the result is ready.
Is that simple? Unfortunately no.
Not to mention the difficulties with the right choice of a test malware, the most difficult thing here is that testing “as close as possible to the combat” conditions requires these very close conditions that are extremely difficult to automate. Those. such tests require VERY much manual mechanical work.
And for this reason, such tests are carried out extremely rarely and in a very truncated mode. In the previous post I have already listed a few. But even here, each has its own characteristics and nuances.
In short, how to properly advise testing correctly is clear and obvious to me. But where to find madmen who could take on [free of charge] carrying out such test measurements is unknown to me. So, alas, to look at the correct results of the right tests - there is no place so far ...
Here is such a B flat minor at the end of this music composition: (
As expected, the recent post about the Passmark performance test made some noise. And the main thing - the noise was not so much in the blog, as in the antimalvarnoy industry. Well, this is exactly what the real Montmorency usually wants :)
In short - it turned out. Because I want to find the truth. And not only that - I would very much like to somehow stimulate the correctness of antivirus testing.
')
And how do you need to test antiviruses?
Yeah, so firstly. In order to avoid raids. To immediately bring down komenty, that type we will now teach testers to cook stool according to our recipes, so that our products climbed into the lead. There is an answer to this - we rarely do NOT fall into the top three in various tests anyway. And secondly, the fact that what is being discussed now is not my invention, but the AMTSO industrial standards (Anti-Malware Testing Standards Organization is such an organization), in which there are representatives of almost all leading AV-vendors and various well-known experts. .
Alas, the general public and the progressive world community know little about AMTSO, and most testers still prefer to do their work in the old manner. It is understandable - it is cheap and familiar, and the user, they say, the main thing is the result - who took the first, second, third place, and who went to the dump.
It seems everyone is happy, BUT! It really distorts the picture. In the end, it turns out that not the best software creeps upward, the position in the list of test winners practically does not correspond to the real level of protection provided - in short, bullshit and uh ... consumer fraud, in short.
Why am I so hot — sometimes it is just a shame that time and resources are spent not to “do business” —but to provide demonstrations in the next shit testing. To show the result is not worse than those that stupidly sharpens their products not on real quality - but only to test more correctly the rest.
Here it is. And now - let's get to the main point.
So, how not to test.
Classic, good old- on-demand test .
Actually, this is the most common standard and familiar test, and once, long ago, in pre-massively Internet times, it really was the most correct one.
// we, by the way, in 1994 staged the international premiere of AVP just on such a test of the Hamburg University - we participated for the first time and immediately broke everyone.
The testing technique is as follows: the disk is taken and scored by Malvara, the more - the better, the most varied, to which the arms have reached. Then various anti-virus scanners are set on him and the number of detectors is measured. Cheap and angry. But already 10 years old as absolutely irrelevant !
Why? And because anti-virus signature, heuristic and other “scanning” engines are only part of a set of technologies that are used in the real world for protection! (and the value of these engines in the general level of protection is rapidly falling). Often, the scanner generally works as a last resort for purely surgical work: for example, System Watcher tracks some Trojan, understands the infection pattern, and then transfers the task of ejection to the scanner.
Another drawback is the base for Malvari to scan.
There are two extremes and both vicious. Too few malware files - you know, irrelevant. Too many samples - the same trabla, but from the other end: too much garbage gets into mega-collections (broken files, data files, not-so-good malware, for example, scripts that use malware, etc.) and clean collection from such garbage - the hardest and ungrateful work. In addition, low paid :)
And the main drawback - under such tests, you can sharpen any product. To show outstanding results in these tests. Fitting the product to a test is done elementary - you just need to detect the files that are used in the test to the maximum. Do you follow the train of thought?
Again.
In order to show in the "scan tests" close to 100% result, do not rest against and improve the quality of technology. You just have to detect everything that gets into these tests. Something like a joke about hunters and bears:
- But the bear runs faster than us! - says the first hunter.
“And I don’t need to run faster than a bear,” answers the second, “I just need to run faster than you.”
To win these tests do not need to "run faster than a bear." You just need to cling to the sources of Malvari, which use the most famous testers (and these sources are known - VirusTotal, Jotti and Malvaro-exchangers of anti-virus companies) - and stupidly detect everything that others detect. Those. if the file is detected by competitors - just detect it on MD5 or something like that.
Everything! I am personally ready from scratch , by a couple of developers , to make a scanner that will show almost 100% of detections in a couple of months. // why exactly two developers are needed - just in case, suddenly one gets sick.
In short, on-demand tests are incorrect tests. Since they do not show anything real, it is easy to adjust to them and it is extremely difficult to win honestly.
How can you test, but where NOT SPECIAL is better not to look.
There is a zoo of all kinds of niche tests that show the quality of antiviruses in some very specific area .
In principle, they have the right to exist; moreover, they are extremely useful for comparing the quality of production of any particular feature. But it is necessary to write in BIG letters about this specifics and that the test does not take into account all the abilities of the product . What the average user has nothing to do here is purely industrial tests that carry a payload only for specialists and other geeks.
For example, a test for treating a specific case (how a product copes with treating a system that is infected in a certain way), a test for false positives (like the product "Falsit" on pure software), a proactive test (how the product catches malware without signatures, i.e. proactive technologies), on-access test (quality of on-access scanner for real-time operations with malware), measurements of interface performance and gagging , etc. - various tests for a specific purpose.
The type of individual tests for acceleration, deceleration, and gasoline consumption - but only separately, without reference to each other. Since the best result can show an absolutely non-usable product in everyday life, such as a Formula 1 car :)
Finally, how to test and where to look ...
I'll start from afar. What are tests for?
First of all, to show the quality of protection. Yes, with all equal usability and performance are also important, but, ultimately, we are “to go, not checkers.” Therefore so.
How to check the quality of protection?
Of course, in an environment as close as possible to combat ! The methodology of the correct test should be based on the most common scenarios for users in real life . All the rest is a compromise and formalities from the category “if it were yes, if it were so.”
For this task, the dynamic or real world test is best suited.
The meaning is simple: we take an ordinary computer, put an antivirus on it with the default settings and try to launch the current malware in every possible way. It's simple!
The product works at full capacity, the situation is as close as possible to the combat! And the user as a result receives the correct measurement of quality of an antivirus and the relevant information for a reasonable choice. Then measure the load on the system, the size of updates, add the price, put “weight per points” for each category of testing - and the result is ready.
Is that simple? Unfortunately no.
Not to mention the difficulties with the right choice of a test malware, the most difficult thing here is that testing “as close as possible to the combat” conditions requires these very close conditions that are extremely difficult to automate. Those. such tests require VERY much manual mechanical work.
And for this reason, such tests are carried out extremely rarely and in a very truncated mode. In the previous post I have already listed a few. But even here, each has its own characteristics and nuances.
In short, how to properly advise testing correctly is clear and obvious to me. But where to find madmen who could take on [free of charge] carrying out such test measurements is unknown to me. So, alas, to look at the correct results of the right tests - there is no place so far ...
Here is such a B flat minor at the end of this music composition: (
Source: https://habr.com/ru/post/131110/
All Articles