Wow, looks like we've gone full circle.

I remember my anti-virus software on my Win 3.1/DOS box detecting viruses with both signatures and heuristics. It was fairly decent at it too. Of course back then anti virus was actually a technical challenge what with viruses deploying stealth techniques like polymorphism, IAT hooking etc rather than an exercise in generating hashes of files.

The proposal is a step in the right direction - but it is not good enough. If the tests are conducted as the proposal suggests, they will essentially test obsolete (months-old) heuristic scanning capability. This isn't good enough, although it's better than the current situation.

The proper way to do it is to install an up-to-date version of the AV software and attack it with *live* malware. Don't just let it scan a "dead" collection of samples. Start executing the malware - and then see if *any* part of the package manages to prevent (completely!) the infection of the test machine. It doesn't matter whether this is the scanner, the heuristic analyzer, the behavior blocker, or anything else that stops the malware - it's sufficient if the malware is stopped.

If you ask me - this is what is wrong with the antivirus / endpoint security industry today. Too many people patting themselves on the back for fighting malware, and not attention paid to real-world effectiveness. This article just sent me off on a rant - http://bit9.com/blog/home/tabid/15398/bid/2456/Antivirus-Protecting-Against-Yesterday-s-Malware.aspx

Is it not possible to mate, for the purposes of testing, an old or even blank/minimal signature file with the latest heuristic engine if you want to test the capacity of the heuristics to detect threats?

They want to test the ability to protect against previously unknown threats.

The best way to obtain test against unknown threats would bet to travel one week into the future and obtain the latest real world nasties.

However until they get their time travel machine working, they decided to do the next best thing.

Today's threats vs AV software that has been frozen in time for a week.

As far as it being unfair because the AV software doesn't have the latest updates, I wish I lived in a world where AV software became dramatically more effective on a week to week basis. :)

Anonymous Coward: Yes, theoretically, it is possible. In practice, however, it is not. First of all, practically no AV vendor will supply you with a "blank signature file". We (F-PROT) used to do it only for our macro malware signatures and nowadays even we don't do it any more. Furthermore, the term "signature" is misleading. Contrary to popular belief, it's not a collection of scan strings for known malware. Nowadays it is a complex database containing whole programs for detecting malware. Often even the scanning engine of the AV product is updated by this database. So, if you use an old database, you're running the risk of using an old (even buggy) AV engine.