Feeds

Security vendors cry foul over exploit tests

'Like testing ABS brakes by pushing a car over a cliff'

Choosing a cloud hosting partner with confidence

Anti-malware vendors have launched a counter-attack on a study questioning the effectiveness of internet security suites, suggesting that the methodology in tests carried out by vulnerability notification firm Secunia was fundamentally flawed.

As previously reported, Secunia tested a selection of 12 internet security suites against how well they did at blocking exploits. None came out of this particular well, with the highest scorer, Symantec, thwarting only 64 out of 300 exploits.

Firms whose products featured in the tests, including Panda Security, cried foul, saying the tests only looked at one of a battery of defensive measures their suites offer. Independent testing organisation AV-test.org backed this line in criticising the tests as focusing purely of on-demand scanning of potentially malicious files. Meanwhile, security firms not involved in the Secunia's bake-off, such as Sunbelt Software, also waded in to cry foul, decrying the exercise as a publicity stunt.

Thomas Kristensen, chief technology officer at Secunia, responded to this criticism by saying vendors had misunderstood the purpose of the tests. While acknowledging its tests weren't comprehensive, Kristensen argued that they illustrated the importance of patching vulnerable applications and adopting a "defense in depth" approach to defending against hackers. He said users shouldn't be lulled into the belief that simply installing and updating internet security suites was enough.

"We only tested one specific aspect (exploitation of vulnerabilities) because too many users believe (and are lead to believe by the marketing material) that they only need a security suite to protect them against various threats including hackers," Kristensen told El Reg.

"Our point is not that Internet Security Suites are useless (they are quite useful for most users). Instead, our point is that they protect insufficiently against hackers and that it is better to prevent attacks by patching rather than relying on other security measures alone".

Panda Security virus analysts Pedro Bustamante compares the Secunia tests to testing a car’s ABS breaking systems by "throwing it down a 200 meter cliff" in a passionate, but nonetheless technically illuminating, blog posting.

"If you only test one part of a product against exploits, which by the way is the part of the product which is not designed to deal with exploits, and leave out of the test the part of the product that DOES deal with exploits and vulnerabilities, there's a very good chance the results will be misleading," Bustamante writes.

"Internet Security Suites do not rely on signature detection alone since many years ago. Panda's (and other) products integrate behavioral analysis, context-based heuristics, security policies, vulnerability detection, etc. However none of these technologies were tested by Secunia."

Bustamante says a number of exploits listed as not detected by Panda are actually blocked if any attempt is made to run them.

Kristensen responded: "It seems quite odd that the AV-vendors are so busy claiming that they can detect literally anything malicious when executed. If they can do that, why do they then have to push "signature" updates to their software so frequently?

"It is obviously much better to be able to detect malicious content while it is passive instead of relying on (hopefully) being able to catch it once executed," he added.

Secunia has taken some of the substantive points made by Panda on board while defending itself against suggestions that its test might have been unfair. "We find the criticism from Panda useful and if we do conduct another test of the file-based test cases, then we will categorise their performance into: Unzipping, manual scan, and opening of test case with vulnerable application," Kristensen said.

Although Secunia and security vendors are at loggerheads over the implications of the tests, there's general agreement that patching is a key element in keeping systems secure - a point that, if nothing else, Secunia's tests have amply illustrated. ®

Internet Security Threat Report 2014

More from The Register

next story
FYI: OS X Yosemite's Spotlight tells Apple EVERYTHING you're looking for
It's on by default – didn't you read the small print?
Russian hackers exploit 'Sandworm' bug 'to spy on NATO, EU PCs'
Fix imminent from Microsoft for Vista, Server 2008, other stuff
Edward who? GCHQ boss dodges Snowden topic during last speech
UK spies would rather 'walk' than do 'mass surveillance'
Microsoft pulls another dodgy patch
Redmond makes a hash of hashing add-on
'LulzSec leader Aush0k' found to be naughty boy not worthy of jail
15 months home detention leaves egg on feds' faces as they grab for more power
China is ALREADY spying on Apple iCloud users, claims watchdog
Attack harvests users' info at iPhone 6 launch
Carders punch holes through Staples
Investigation launched into East Coast stores
prev story

Whitepapers

Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Why and how to choose the right cloud vendor
The benefits of cloud-based storage in your processes. Eliminate onsite, disk-based backup and archiving in favor of cloud-based data protection.
Three 1TB solid state scorchers up for grabs
Big SSDs can be expensive but think big and think free because you could be the lucky winner of one of three 1TB Samsung SSD 840 EVO drives that we’re giving away worth over £300 apiece.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.