Original URL: https://www.theregister.com/2010/08/09/anti_virus_effectiveness_analysis/
Anti-virus defences even shakier than feared
Security firms attack 'flawed' tests
Posted in Security, 9th August 2010 15:48 GMT
Updated Anti-virus technologies may be even more ineffective than feared, if a controversial new study is to be believed.
A study by web intelligence firm Cyveillance found that, on average, vendors detect less than 19 per cent of malware attacks on the first day malware appears in the wild. Even after 30 days, detection rates improved to just 61.7 per cent, on average.
Anti-virus vendors have criticised the methodology of the study as hopelessly flawed not least because it only looked at signature-based detection of malware.
Cyveillance argues its research shows that users ought to practice safe computing - such as avoiding unknown or disreputable websites and increasing security settings on their web browser - as a way of minimising security risks rather than relying on up to data anti-virus to protect them.
The security research outfit criticises "signature-based" anti-virus technologies; which is fair enough but rather ignores the point that vendors have long adopted generic detection of malware strains, and are introducing crowd-based architectures as a means of providing protection from the ever-increasing volume of malware threats.
Luis Corrons, technical director of Panda Security, which was not tested as part of the research, said that Cyveillance had only tested one component of anti-malware protection. The tests ignored anything except anti-virus signatures - despite the fact this is only one layer of the protection offered by modern security / anti-malware suites.
"As far as I’ve seen, they have only tested static signature detection capabilities, Corrons told El Reg. "This is the very first technology ever implemented in an antivirus."
"It is good to detect known malware, but it is clearly not enough, and every serious antivirus vendor knows that, and even some of us recognize it in public, Panda has been saying this for years. That’s why most of the major vendors have been developing proactive technologies: behavior analysis/blocking, cloud-based detections, etc.," Corrons concluded.
David Harley, senior research fellow at anti-virus firm Eset, which was tested, said Cyveillance conclusions that anti-virus solutions alone do not adequately protect individuals and enterprises is reasonable but its test methodology is flawed. For one thing Cyveillance looked at just 1,708 samples, a minute fraction of the tens of thousands of malicious binaries that pass through anti-virus labs every day.
"You can't convincingly claim statistical precision with a data set of 1,708 samples, Harley explained. "You certainly can't rank comparative performance meaningfully on that basis unless you can demonstrate accurate weighting for prevalence, and there's no indication of that in the report. Harley said Cyveillance may have looked at on-access scanner performance and failed to carry out "true dynamic or whole product testing", repeating a problem of other tests where "testers draw big conclusions from tests that only look at a single detection behaviour". "Anti-virus products miss a lot of malware.However, the exact number or proportion of threats missed is a bit harder to calculate (or even guess at) than Cyveillance seems to think," Harley concluded. ®