Original URL: http://www.theregister.co.uk/2013/01/17/avtest_microsoft_test_dispute/

AV-Test boss dismisses Microsoft criticism of malware test results

Says Redmond's not picking up the fresh threats

By Iain Thomson

Posted in Security, 17th January 2013 21:16 GMT

A war of words has broken out over security testing, with Microsoft and the AV-Test Institute going head-to-head over Redmond's failure to qualify for the last round of certification from the German testers.

On Tuesday, AV-Test announced its December round of security software evaluations, and both Microsoft's Security Essentials and its Forefront suite failed to achieve certification for the second month running. The next day, Microsoft questioned the results in an extended blog post by Joe Blackbird, program manager at Microsoft's Malware Protection Center.

"Our review showed that 0.0033 percent of our Microsoft Security Essentials and Microsoft Forefront Endpoint Protection customers were impacted by malware samples not detected during the test," Blackbird said. "In addition, 94 percent of the malware samples not detected during the test didn't impact our customers."

But Andreas Marx, CEO of AV-Test, told El Reg that one would expect this. "I'm not surprised that in testing two months later they get different results," he said.

The AV-Test results show that Microsoft's twin security programs protected against 100 per cent of known threats, as did every other security suite. The two packages produce low rates of false positives in comparison to the competition and are significantly lighter on processor load during operations.

But where Redmond is falling down is in protecting against zero-day attacks. Security Essentials and Forefront both scored last in this regard among all the suites tested, getting 78 per cent of zero-days apiece. Blackbird said that AV-Test attached too much importance to the zero-day threat in its metrics, since that section of the testing accounts for 50 per cent of the final score, but Marx argued that zero-day performance was crucial to real-world threats.

He explained that many security tests rely on the "wild list" of malware in circulation among the community, but that the samples can be 80-90 days old in some cases and will therefore have been supplanted by newer varieties of core malware routines. As such, testing against current threats is key to the institute's metric.

AV-Test teams take malware that is minutes old, Marx explained, and run the data into the security testing suite. A testing process carried out by Microsoft much later would be bound to cover the malware tested, since samples would already have been reported.

"Today, every two seconds we see three new malware samples, which are summing up to a few million samples per month. Instead of looking at millions of samples, our focus is on the unique families," Marx explained.

"Out of every family, we select recent samples in order to use them in our tests. So the impact of these samples is indeed low, however, the impact of the malware family is considerably high."

Overall AV-Tests did show that the effectiveness of antivirus code has declined. With this latest batch of tests, the average zero-day malware blocking success rate was 92 per cent. That doesn't sound too bad, but given the colossal amount of crooked code out there, it still represents a big problem – not to mention the ever-present threat of completely unknown families of malware coming online.

BitDefender, Kaspersky, and Norton topped December's consumer security rankings from AV-Test, with Kaspersky, F-Secure, and Sophos getting the best scores for business protection. ®