Feeds

Security vendors cry foul over exploit tests

'Like testing ABS brakes by pushing a car over a cliff'

Beginner's guide to SSL certificates

Anti-malware vendors have launched a counter-attack on a study questioning the effectiveness of internet security suites, suggesting that the methodology in tests carried out by vulnerability notification firm Secunia was fundamentally flawed.

As previously reported, Secunia tested a selection of 12 internet security suites against how well they did at blocking exploits. None came out of this particular well, with the highest scorer, Symantec, thwarting only 64 out of 300 exploits.

Firms whose products featured in the tests, including Panda Security, cried foul, saying the tests only looked at one of a battery of defensive measures their suites offer. Independent testing organisation AV-test.org backed this line in criticising the tests as focusing purely of on-demand scanning of potentially malicious files. Meanwhile, security firms not involved in the Secunia's bake-off, such as Sunbelt Software, also waded in to cry foul, decrying the exercise as a publicity stunt.

Thomas Kristensen, chief technology officer at Secunia, responded to this criticism by saying vendors had misunderstood the purpose of the tests. While acknowledging its tests weren't comprehensive, Kristensen argued that they illustrated the importance of patching vulnerable applications and adopting a "defense in depth" approach to defending against hackers. He said users shouldn't be lulled into the belief that simply installing and updating internet security suites was enough.

"We only tested one specific aspect (exploitation of vulnerabilities) because too many users believe (and are lead to believe by the marketing material) that they only need a security suite to protect them against various threats including hackers," Kristensen told El Reg.

"Our point is not that Internet Security Suites are useless (they are quite useful for most users). Instead, our point is that they protect insufficiently against hackers and that it is better to prevent attacks by patching rather than relying on other security measures alone".

Panda Security virus analysts Pedro Bustamante compares the Secunia tests to testing a car’s ABS breaking systems by "throwing it down a 200 meter cliff" in a passionate, but nonetheless technically illuminating, blog posting.

"If you only test one part of a product against exploits, which by the way is the part of the product which is not designed to deal with exploits, and leave out of the test the part of the product that DOES deal with exploits and vulnerabilities, there's a very good chance the results will be misleading," Bustamante writes.

"Internet Security Suites do not rely on signature detection alone since many years ago. Panda's (and other) products integrate behavioral analysis, context-based heuristics, security policies, vulnerability detection, etc. However none of these technologies were tested by Secunia."

Bustamante says a number of exploits listed as not detected by Panda are actually blocked if any attempt is made to run them.

Kristensen responded: "It seems quite odd that the AV-vendors are so busy claiming that they can detect literally anything malicious when executed. If they can do that, why do they then have to push "signature" updates to their software so frequently?

"It is obviously much better to be able to detect malicious content while it is passive instead of relying on (hopefully) being able to catch it once executed," he added.

Secunia has taken some of the substantive points made by Panda on board while defending itself against suggestions that its test might have been unfair. "We find the criticism from Panda useful and if we do conduct another test of the file-based test cases, then we will categorise their performance into: Unzipping, manual scan, and opening of test case with vulnerable application," Kristensen said.

Although Secunia and security vendors are at loggerheads over the implications of the tests, there's general agreement that patching is a key element in keeping systems secure - a point that, if nothing else, Secunia's tests have amply illustrated. ®

Choosing a cloud hosting partner with confidence

More from The Register

next story
SMASH the Bash bug! Apple and Red Hat scramble for patch batches
'Applying multiple security updates is extremely difficult'
Apple's new iPhone 6 vulnerable to last year's TouchID fingerprint hack
But unsophisticated thieves need not attempt this trick
Hackers thrash Bash Shellshock bug: World races to cover hole
Update your gear now to avoid early attacks hitting the web
Oracle SHELLSHOCKER - data titan lists unpatchables
Database kingpin lists 32 products that can't be patched (yet) as GNU fixes second vuln
Who.is does the Harlem Shake
Blame it on LOLing XSS terroristas
Researchers tell black hats: 'YOU'RE SOOO PREDICTABLE'
Want to register that domain? We're way ahead of you.
Stunned by Shellshock Bash bug? Patch all you can – or be punished
UK data watchdog rolls up its sleeves, polishes truncheon
Ello? ello? ello?: Facebook challenger in DDoS KNOCKOUT
Gets back up again after half an hour though
prev story

Whitepapers

Providing a secure and efficient Helpdesk
A single remote control platform for user support is be key to providing an efficient helpdesk. Retain full control over the way in which screen and keystroke data is transmitted.
Intelligent flash storage arrays
Tegile Intelligent Storage Arrays with IntelliFlash helps IT boost storage utilization and effciency while delivering unmatched storage savings and performance.
Beginner's guide to SSL certificates
De-mystify the technology involved and give you the information you need to make the best decision when considering your online security options.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.
Secure remote control for conventional and virtual desktops
Balancing user privacy and privileged access, in accordance with compliance frameworks and legislation. Evaluating any potential remote control choice.