Feeds

Security vendors cry foul over exploit tests

'Like testing ABS brakes by pushing a car over a cliff'

Seven Steps to Software Security

Anti-malware vendors have launched a counter-attack on a study questioning the effectiveness of internet security suites, suggesting that the methodology in tests carried out by vulnerability notification firm Secunia was fundamentally flawed.

As previously reported, Secunia tested a selection of 12 internet security suites against how well they did at blocking exploits. None came out of this particular well, with the highest scorer, Symantec, thwarting only 64 out of 300 exploits.

Firms whose products featured in the tests, including Panda Security, cried foul, saying the tests only looked at one of a battery of defensive measures their suites offer. Independent testing organisation AV-test.org backed this line in criticising the tests as focusing purely of on-demand scanning of potentially malicious files. Meanwhile, security firms not involved in the Secunia's bake-off, such as Sunbelt Software, also waded in to cry foul, decrying the exercise as a publicity stunt.

Thomas Kristensen, chief technology officer at Secunia, responded to this criticism by saying vendors had misunderstood the purpose of the tests. While acknowledging its tests weren't comprehensive, Kristensen argued that they illustrated the importance of patching vulnerable applications and adopting a "defense in depth" approach to defending against hackers. He said users shouldn't be lulled into the belief that simply installing and updating internet security suites was enough.

"We only tested one specific aspect (exploitation of vulnerabilities) because too many users believe (and are lead to believe by the marketing material) that they only need a security suite to protect them against various threats including hackers," Kristensen told El Reg.

"Our point is not that Internet Security Suites are useless (they are quite useful for most users). Instead, our point is that they protect insufficiently against hackers and that it is better to prevent attacks by patching rather than relying on other security measures alone".

Panda Security virus analysts Pedro Bustamante compares the Secunia tests to testing a car’s ABS breaking systems by "throwing it down a 200 meter cliff" in a passionate, but nonetheless technically illuminating, blog posting.

"If you only test one part of a product against exploits, which by the way is the part of the product which is not designed to deal with exploits, and leave out of the test the part of the product that DOES deal with exploits and vulnerabilities, there's a very good chance the results will be misleading," Bustamante writes.

"Internet Security Suites do not rely on signature detection alone since many years ago. Panda's (and other) products integrate behavioral analysis, context-based heuristics, security policies, vulnerability detection, etc. However none of these technologies were tested by Secunia."

Bustamante says a number of exploits listed as not detected by Panda are actually blocked if any attempt is made to run them.

Kristensen responded: "It seems quite odd that the AV-vendors are so busy claiming that they can detect literally anything malicious when executed. If they can do that, why do they then have to push "signature" updates to their software so frequently?

"It is obviously much better to be able to detect malicious content while it is passive instead of relying on (hopefully) being able to catch it once executed," he added.

Secunia has taken some of the substantive points made by Panda on board while defending itself against suggestions that its test might have been unfair. "We find the criticism from Panda useful and if we do conduct another test of the file-based test cases, then we will categorise their performance into: Unzipping, manual scan, and opening of test case with vulnerable application," Kristensen said.

Although Secunia and security vendors are at loggerheads over the implications of the tests, there's general agreement that patching is a key element in keeping systems secure - a point that, if nothing else, Secunia's tests have amply illustrated. ®

Mobile application security vulnerability report

More from The Register

next story
Yorkshire cops fail to grasp principle behind BT Fon Wi-Fi network
'Prevent people that are passing by to hook up to your network', pleads plod
HIDDEN packet sniffer spy tech in MILLIONS of iPhones, iPads – expert
Don't panic though – Apple's backdoor is not wide open to all, guru tells us
NEW, SINISTER web tracking tech fingerprints your computer by making it draw
Have you been on YouPorn lately, perhaps? White House website?
LibreSSL RNG bug fix: What's all the forking fuss about, ask devs
Blow to bit-spitter 'tis but a flesh wound, claim team
Black Hat anti-Tor talk smashed by lawyers' wrecking ball
Unmasking hidden users is too hot for Carnegie-Mellon
Attackers raid SWISS BANKS with DNS and malware bombs
'Retefe' trojan uses clever spin on old attacks to grant total control of bank accounts
Manic malware Mayhem spreads through Linux, FreeBSD web servers
And how Google could cripple infection rate in a second
Mozilla fixes CRITICAL security holes in Firefox, urges v31 upgrade
Misc memory hazards 'could be exploited' - and guess what, one's a Javascript vuln
prev story

Whitepapers

Designing a Defense for Mobile Applications
Learn about the various considerations for defending mobile applications - from the application architecture itself to the myriad testing technologies.
How modern custom applications can spur business growth
Learn how to create, deploy and manage custom applications without consuming or expanding the need for scarce, expensive IT resources.
Reducing security risks from open source software
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Consolidation: the foundation for IT and business transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.