Comparative testing of 15 antivirus programs in the new S'T magazine

At Security @ Interop, an event completely unjustly overlooked by many security guards, it was possible to take the pilot (April) issue of C'T-Russia magazine , which I gladly did. Now he walks the hands of the whole company.

C'T found a very interesting and thoughtful comparative testing of 15 antivirus programs. Hands finally reached the "processing" of this very "tasty" content. I will give here only the introductory part about the methodology and conclusions. This is exactly the same will be a comprehensive analysis, on the basis of which recently fashionable (and now you can) curse or praise products of different vendors.

I believe that for a team of a new computer technology magazine, conducting such a comprehensive test is a big bid for leadership, because the tests are no worse than those conducted by Neil Rubenking from PC Magazine USA. Now there is something to read in Russia - a hand-made test, described in human language and impartially filed, which together is a “trick” of C'T. It is strongly recommended to adopt Sergey Ilyin from Anti-Malware.ru.
')
I also liked the words of the editor-in-chief about the audience and the mission of the magazine (quote):
We do not do this nonsense. We make the most professional computer magazine in Russia ... for those who need honest and truthful information.

Respect Pavel Shoshin, author of the review, and Glavred Andrei Kokourov, who moved from CHIP. I hadn’t had such confidence in their professionalism before, but as it turned out, at CHIP, everything was decided by the “easy reading” format, which recently too many media outlets and test labs are putting pressure on.

In general, the praises of the magazine, I otpele, you can go on to read. :)

Big hunt

Trojans, spyware, bots - everything seems to be the same here, but, nevertheless, it is already different: every hour more than a thousand newcomers replenish the virus “zoo” of antivirus laboratories. And judging by the results of testing 15 antivirus programs, an increasing majority of them slip through the antivirus protection.

Along with the antivirus industry’s luminaries, some exotic programs passed our tests: Avast Antivirus Pro, AVG Anti-Malware, Avira AntiVir Personal Edition Premium, BitDefender, CA Antivirus Plus, ClamWin, Dr.Web, F-Secure Antivirus, McAfee VirusScan, Microsoft OneCare and Trend Micro Antivirus + Antispyware.

The main mechanism for detecting malicious programs is still the search for files by known strings of characters. The quality of such signature recognition is determined by checking the set of ITW viruses (from the English. In The Wild - viruses that pose a real danger to users at the time of testing). However, this approach did not become dominant, if only because it does not represent the most active class of malicious programs at the moment - Trojans. In addition, this list of viruses is not replenished as often, constantly becoming outdated, and the list itself, consisting of approximately 1,400 copies, is too small. Therefore, ITW testing in this case was carried out only for the sake of completeness of the experiment, and its results were not taken into account when summing up the results.

The fact that avast! and Kaspersky Anti-Virus did not prove to be the best, although during an on-demand scan all viruses from the ITW kit were detected, and on the fly both packages skipped over one file, indicating problems with the quality of the scan. But ClamWin and Dr.Web made more significant mistakes, missing 25 and 7 viruses, respectively.

In the case of our own testing, antivirus programs should have checked the virus database, which has more than a million worms, backdoors, bots and trojans. In this case, only those malicious codes that were actually active in the past six months were used; extinct dinosaurs since DOS and Windows 95 are not counted.

When testing, it was desirable to get the highest percentage of recognition, but we understood that none of the scanning methods could detect absolutely all viruses. In particular, the Avira product approached the ideal with an impressive result - over 99% of recognition. The result of 95%, necessary for a positive assessment, received three more products: Avast !, AVG and BitDefender. In addition to ClamWin and Dr.Web, which, as already mentioned, did not perform well in this test, CA Antivirus looked weak with 55% - clearly not enough for reliable protection.

Quite good results were obtained in identifying adware and spyware, the test base of which consisted of 25,000 programs. If earlier anti-spyware tools were needed to combat it, now developers include data from this category of malware in signatures. We were not mistaken, the term “malicious” is quite appropriate here: utilities that can capture sensitive information or make lists of users' interests and send them over the network deserve such a title. By the way, the programs that did well with the viral “zoo” also performed well in tests with detecting spyware. CA Antivirus and Norton Antivirus failed in this test.

A positive point in the overall picture was that the developers increased the frequency of software updates, as well as the speed of response to new virus threats. Softwin and Kaspersky Anti-Virus are leading in this respect. Only CA Antivirus, McAfee, and Microsoft users are forced to wait for updated signatures on average more than 12 hours compared with the fastest competitor. But the updates from Microsoft One Care were late even more than a day.

I can see what you do not see
The reason that good signature recognition results, as well as the short intervals between updates are not enough to reliably protect a computer, can be shown with a simple calculation. Let's say the developer will have time within an hour after the appearance of the new Trojan program to create the necessary signatures, check them and send them to their customers. However, at this time, malware is distributed through a network of bots using about 10,000 active zombie computers. Each of them can send an e-mail message (that is, more than a thousand per hour) within three seconds. As a result, the owner of a network of bots will have time to reach about 10 million of his victims before the first signature appears. Of course, some figures can be challenged, but the general principle is clear: in case of an epidemic, signatures are always late.

To create programs that can detect previously unknown pests, developers began to use heuristic methods. They allow, in particular, to detect typical code sequences or issue warnings if the scanned program tries to load the resident module into memory.

The antivirus software we are testing should have detected new viruses with the help of old signatures.

Behavioral analyzer testing

Behavioral analysis (proactive defense) differs from signature-based and heuristic methods, which serve only to recognize the malicious files themselves. The difference in testing begins with a selection of viruses that none of the tested packages can identify using signatures. When launching, the virus should get into the same environment every time, and, for example, be able to download its components from the network, despite the fact that the corresponding servers at the time of testing could cease to exist.

During the testing itself, the virus is given the opportunity to work and monitor its behavior, as well as how the antivirus program behaves. If the protection proposes to interrupt the work, then the testing is terminated. After that, they check whether the virus managed to penetrate and cause damage, or its attack was successfully repelled, and the open files and registration keys were deleted.

In total, twelve viruses that pose a danger to Windows (Spy.VB.QJ, Packer, DNS-Changer.OL, Rbot.BMR, Hmir.DK, Delf.FYR, IRCBot.CHR, Agent.CDM, RBot.XKW , PcClient.BAL, Pakes.AKT, Zlob.KF). When evaluating the program received one point, if as a result the computer was not subjected to infection, that is, the virus could not install the executable components. If the antivirus software managed that the system was not infected at the next launch, the antivirus received a half point.

The method of recognition by behavior is prone to failures, especially if it is used only to issue messages about certain actions that occur during normal operation. Therefore, on the basis of the installation and updating of ten standard programs like ICQ, Winamp or Microsoft Office, we also checked the proportion of false positives.

Saving anchor
The fact that signature recognition methods do not provide reliable protection, and heuristic analysis does not have high reliability, has been known for several years. The solution to this problem is also known: when monitoring a system, the antivirus must also recognize previously unknown malware by their behavior. With an increase in suspicious activity, he warns the user and offers to stop tasks that could harm the computer, or even cancel some previously performed operations. This feature, called “Proactive Defense” (Proactive Guard, behavioral analyzer, may also be called the Behavioural Blocker) is also found in the description of many software products.

Our testing showed rather disappointing results. Only F-Secure Deepguard of F-Secure, which recognized all 12 “pests”, showed itself well, and in most cases it managed to prevent the infection of the computer. In addition to it, Kaspersky Anti-Virus and BitDefender also managed to detect malicious programs by their behavior. However, they could not completely prevent the infection of the system.

Development companies McAfee, Norton, Microsoft, and Trend Micro follow the path of least resistance and offer to evaluate the behavior of suspicious programs by the users themselves. When using these antivirus vendors, you will often come across messages that were often encountered at the dawn of software firewalls: “The XYZ program wants to perform some action - do you want to allow this?”. The catch is this: these messages almost always relate to individual actions, such as adding an autorun point to the registry. Even if the user forbids this action, the virus will not be neutralized yet; some of its components may still remain active. In addition, such warnings often appear when installing harmless programs. Well disguised as a video codec, a trojan in this way can hardly be detected, since the user himself wants to install this program on his computer.

These results are confirmed by our experiments with simple keyloggers created from existing code fragments, which invisible to the user save the data entered from the keyboard. One of the keyloggers was added to the registry as a startup point, the other was installed as a kernel driver, the third as a service, and the latter additionally established a connection to the IRC server. In this case, only F-Secure, BitDefender and Kaspersky Labs showed acceptable results. F-Secure recognized all keyloggers as high-risk applications, which was reflected in the rating. Kaspersky Anti-Virus reported installing suspicious drivers, noticed the creation of new files and their registration as a startup point and as a service. BitDefender's B-Have program recognized all keyloggers with the exception of the fake kernel driver.

The McAfee and Trend Micro programs recorded only the registry changes made by the first keylogger. With NOD32 and OneCare, heuristic analysis revealed only the first keylogger, who entered data into the registry branch responsible for autoloading. Other antiviruses could not detect anything at all.

Peekaboo
Although more and more viruses are using rootkit methods to hide their presence in the system, only a few anti-virus programs are ready for this to a sufficient degree. Only programs from F-Secure, Norton and Panda during the test were able to detect and remove all active rootkits, although we used only widely known and accessible samples. The fact that AVG recognizes only inactive rootkits can in no way be attributed to it as an asset, even taking into account the fact that the program has a separate module for rootkit scan. Microsoft's ClamWin, Dr.Web, McAfee, and OneCare also have little to do with this type of threat.

If a rootkit attack is suspected, the boot rescue disk is of particular importance, thanks to which you can scan the system without loading it. However, these drives are still saving many manufacturers. And if they don’t save, they’re obviously bumbling: Symantec, as before, loads DOS with an anti-virus scanner whose signature is dated 2002, and Panda offers a Linux-based boot disk that provides access to NTFS disks in read-only mode.

Another method used to trick antivirus scanners is to use archive files that they cannot open. At the same time, attackers change the access rights to the content, for example, in a ZIP archive, so that the anti-virus scanner cannot open it, and it could be done with the help of utilities installed on the computer. Thus, the virus can go unnoticed when checking on the mail gateway.

We tested the stability of individual antiviruses to similar methods of masking during testing using 28 specially prepared archive files in the .rar, .zip and .cab format. They could be opened with the help of standard programs, such as Windows ZIP, WinZIP, WinRar or 7zip integrated into Windows. However, only F-Secure detected hidden viruses in all modified archives.

Similarly, it can be made invisible to antiviruses and malicious code that uses security holes in browsers. It has long been known that Internet Explorer completely ignores some code elements of HTML pages. For example, an attacker can fill his exploit with zero bytes so that it differs from the samples in the antivirus database. The trick with adding zero bytes was performed on all antiviruses, with the exception of BitDefender. Other programs issued warnings, but the exploits managed to confuse them.

Vista and Antivirus
All tests were conducted on computers running Windows Vista. This did not cause any difficulties, since most manufacturers have already adapted their programs for it.

Only a new rights model can create some problems due to the fact that the anti-virus scanner must obtain the necessary rights for a full system scan using the User Account Control (UAC) function. Thus, the AVG and CA Anti-Virus programs must be launched manually as an administrator in order to be able to scan absolutely all files. Otherwise, the anti-virus scanner simply skips the folders of other users. F-Secure is still more complicated with antivirus software - the program ignores other people's folders even if you have the necessary permissions. Such a folder can be scanned only if you go to it and allow the action in the UAC window that appears. When using Avast and Dr.Web to perform special operations, the user must enter the administrator password when using a limited account.

Additionally, we tested the software products on computers running Windows XP. At the same time, we found almost no differences between the results of the work of antiviruses in Vista and in XP. However, it is necessary to take into account that when using specific functions, such as proactive defense and rootkit scan, on computers with Windows XP, in some cases, other results can be obtained.

A little bit of everything
To determine how much the antivirus slows down the computer, we carried out two tests: first, the program had to check about 8000 files for viruses with a total volume of 741 MB. To evaluate the performance of the antivirus during on-the-fly scanning, we also copied files on the local hard drive, which in the absence of an antivirus takes 47 seconds.

High speed work showed NOD32; programs such as Avast !, AVG, Antivir, OneCare, Norton, Panda and TrendMicro also did not slow down the system very much. Significantly worse results were shown by F-Secure, which, with four anti-virus engines, significantly slowed down the work, as well as Kaspersky Anti-Virus, which significantly increased the waiting time when copying file folders.

Most antiviruses quietly monitor the work of mail programs and scan incoming as well as outgoing messages. The presence of this function should not be a decisive factor in the anti-virus software, because even if it is absent, the program will still issue a warning when trying to open or save the attached file. In addition, the antivirus will still be powerless when using an encrypted connection.

A serious problem may be the insufficient quality of the source code of antivirus programs. In 2007, virtually all known anti-virus products were found to have critical security holes that cast doubt on the reliability of the entire scope of antivirus software. For our test, we asked manufacturers a question about the measures they take to improve the security of software products. The few answers could not be fully assessed, and it is also impossible to check them for compliance with reality. Therefore, this question is still open.

I do not plan to publish evaluations for each product, also given in the article, but you can evaluate the summary table and conclusions during the course of the article.

Conclusion

NOD32 turned out to be the only testing participant that managed to detect more than two thirds of new malicious programs; BitDefender ranked second with 41%, while the remaining antiviruses recognized only every third threat or even less. In this case, you can hardly talk about reliable protection of your computer. Such poor results can be explained by the fact that malware developers began to pay more attention to optimizing their creations. Of particular concern is the fact that the antivirus during the test could not detect some of the viruses that they successfully found a year ago.
The solution to this problem can be behavioral recognition of malware, which, however, is currently well implemented only in F-Secure's Anti-Virus 2008 program. , , , . F-Secure . BitDefender, , , . .
, , Avira, , NOD32, , , . Avast AVG . , . CE Anti-Virus Plus, , ClamWin, . Dr.Web McAfee .
Norton Antivirus, , Windows OneCare. , , . , Avira AVG , . Vista Windows Defender - Norton Antivirus. , , . Avast, , .







, - «» , Anti-Malware.ru.
, / , :
1 — BitDefender, Avira, F-Secure
2 — NOD32, AVG, Avast!, KAV 7, Trend Micro, Panda 2008
3 — Norton AV 2008, McAfee AV 2008, Dr. Web 4.44, MS OneCare
4 — CA, ClamWin

, ?