Security vendors cry foul over exploit tests
'Like testing ABS brakes by pushing a car over a cliff'
Regcast training : Hyper-V 3.0, VM high availability and disaster recovery
Anti-malware vendors have launched a counter-attack on a study questioning the effectiveness of internet security suites, suggesting that the methodology in tests carried out by vulnerability notification firm Secunia was fundamentally flawed.
As previously reported, Secunia tested a selection of 12 internet security suites against how well they did at blocking exploits. None came out of this particular well, with the highest scorer, Symantec, thwarting only 64 out of 300 exploits.
Firms whose products featured in the tests, including Panda Security, cried foul, saying the tests only looked at one of a battery of defensive measures their suites offer. Independent testing organisation AV-test.org backed this line in criticising the tests as focusing purely of on-demand scanning of potentially malicious files. Meanwhile, security firms not involved in the Secunia's bake-off, such as Sunbelt Software, also waded in to cry foul, decrying the exercise as a publicity stunt.
Thomas Kristensen, chief technology officer at Secunia, responded to this criticism by saying vendors had misunderstood the purpose of the tests. While acknowledging its tests weren't comprehensive, Kristensen argued that they illustrated the importance of patching vulnerable applications and adopting a "defense in depth" approach to defending against hackers. He said users shouldn't be lulled into the belief that simply installing and updating internet security suites was enough.
"We only tested one specific aspect (exploitation of vulnerabilities) because too many users believe (and are lead to believe by the marketing material) that they only need a security suite to protect them against various threats including hackers," Kristensen told El Reg.
"Our point is not that Internet Security Suites are useless (they are quite useful for most users). Instead, our point is that they protect insufficiently against hackers and that it is better to prevent attacks by patching rather than relying on other security measures alone".
Panda Security virus analysts Pedro Bustamante compares the Secunia tests to testing a car’s ABS breaking systems by "throwing it down a 200 meter cliff" in a passionate, but nonetheless technically illuminating, blog posting.
"If you only test one part of a product against exploits, which by the way is the part of the product which is not designed to deal with exploits, and leave out of the test the part of the product that DOES deal with exploits and vulnerabilities, there's a very good chance the results will be misleading," Bustamante writes.
"Internet Security Suites do not rely on signature detection alone since many years ago. Panda's (and other) products integrate behavioral analysis, context-based heuristics, security policies, vulnerability detection, etc. However none of these technologies were tested by Secunia."
Bustamante says a number of exploits listed as not detected by Panda are actually blocked if any attempt is made to run them.
Kristensen responded: "It seems quite odd that the AV-vendors are so busy claiming that they can detect literally anything malicious when executed. If they can do that, why do they then have to push "signature" updates to their software so frequently?
"It is obviously much better to be able to detect malicious content while it is passive instead of relying on (hopefully) being able to catch it once executed," he added.
Secunia has taken some of the substantive points made by Panda on board while defending itself against suggestions that its test might have been unfair. "We find the criticism from Panda useful and if we do conduct another test of the file-based test cases, then we will categorise their performance into: Unzipping, manual scan, and opening of test case with vulnerable application," Kristensen said.
Although Secunia and security vendors are at loggerheads over the implications of the tests, there's general agreement that patching is a key element in keeping systems secure - a point that, if nothing else, Secunia's tests have amply illustrated. ®
COMMENTS
Fools Gold
...catching a disease.
- Or like not having sex with that infected person? Being a little more careful, right?
-- How many viruses do you get in a single day? week? month? year? tell me the truth, and how many would have infected you if you were a bit more careful?
...trivially cheap.
- HDD overhead, not usage
-- So its ok to read a file twice? every file/directory you open, scan scan..?
..by breaking in.
Nice misquote there, I said; you cannot get a virus (99.999999%) of the time without some user action, "press OK, YES INSTALL", I suggested that you are better off being careful/smart about what you click on then be a fool and let you AV pre-click everything for you.
...MAKE SENSE!
I am saying; If you still want to be protected, you are better off using an old PC as a firewall/gateway. Plus you will learn something.
..chip on my shoulder.
You tell me the AV that you use that isn’t slowing your computer down? I find that a hard to believe statement. As inherently you will get some overhead.
Do you agree with the following statement?
"You are better off having a locally installed and always on antivirus suit on you computer. Instead of a fully patched system (free to patch), with a locally available, non intrusive antivirus system (CLAMwin( also free)) and a firewall/gateway/proxy (again free) with built in active scanning and protection / filtering +many more features?(also free)"
My point is that installing AV on your computer is not the most efficient use of your CPU/HDD/RAM, I am saying you gain almost nothing by having an intrusive, aggressive and heavy application. I do agree that using another computer just for security may seem to be OTT but that fact remains that you are better off stopping viruses at the gateway then on your PC.
Running a computer with no AV is like getting a new computer.
Feel free to use AV, feel free to pay (one way or another).
@antivirus is worse then a virus - no, it isn't, that's meaningless hyperbole.
>"Using anti virus products is like driving with your foot on the brakes,"
Or you might compare it to sticking within the speed limit, so that you know your stopping distance is less than the visible distance ahead.
Or you might compare it to having sex with a condom on, to avoid catching a disease.
Or then again, you might just not bother with the stupid and inaccurate metaphors altogether.
>"Think about all the time and wasted CPU/HDD/RAM "
WTF? About 7% cpu or something - and that's only /during/ file access. A few megabytes here and there, when DRAM and HDD space are trivially cheap.
>"You are much better off patching your system and not employing any active aggressive security system."
Right, so you think that patching your system protects you against viruses? You're an idiot. Patching your system protects you against EXPLOITS. Viruses are just plain old executables, and most of them do /not/ get into your system by breaking in.
>"install a simple linux router (smoothwall or similar) "
THINK about what you're saying. You complain about wasting a few cpu cycles and some ram and disk storage - and your so-called solution is to buy a whole new machine? You really need to put down the crack-pipe, because that DOES NOT MAKE SENSE!
>"i rather have a virus once every 3 years and reinstall then 3 years of slow computing"
Well, I'd rather just use a decent AV software that doesn't slow down my computer, but then I'm not an idiot with a chip on my shoulder.
antivirus is worse then a virus....
Using anti virus products is like driving with your foot on the brakes, just in case you might have an accident you will already be half stopped.
Think about all the time and wasted CPU/HDD/RAM used to protect you from something that is fairly unlikely to get you. The only way to get a virus really is from going to dodgey pr0n sites or downloading dodgey applications or opening a dodgey email attachment.
You are much better off patching your system and not employing any active aggressive security system. Be smarter in your habits and reduce the chances of getting infected.
Stop viruses at the gateway/router, install a simple linux router (smoothwall or similar) and offload all that crap from your computer.
Personally i rather have a virus once every 3 years and reinstall then 3 years of slow computing...
IT infrastructure monitoring strategies
Agentless Backup is Not a Myth
Top 10 SIEM implementer’s checklist
Steps to Take Before Choosing a Business Continuity Partner
Requirements Checklist for Choosing a Cloud Backup and Recovery Service Provider
