Anti-spam packages ‘too unreliable’ to certify

Poor performance of anti-spam packages is frustrating attempts by a leading certification vendor to develop benchmarking standards.

ICSA Labs, which runs one of the most important security industry certifications programmes, has recorded disappointing results in its preliminary tests of eight open source and commercial anti-spam packages.

Larry Bridwell, programme manager at ICSA Labs, an independent division within TruSecure, said it had "trouble getting up to 60 per cent" recognition of spam email in the products it has tested thus far.

ICSA Labs is well known for testing a wide variety of security products - anti-virus, cryptography, PKI, IPSec, VPN, firewall, intrusion detection and content filtering - against specific criteria for compliance and reliability. Products which pass the test - and most do not at the first attempt - earn the right to bear the ICSA Labs Certification Seal. Further testing is carried out to make sure products continue to meet certification standards.

Plans to create a certification programme for anti-spam products flowed out of ICSA's work in certifying content filtering packages.

George Japak, VP of ICSA Labs, told us that in the "near term there's unlikely to be a formal certification programme" for anti-spam products.

"It's hard to say 60 to 70 per cent is good enough," he said.

However stakeholders in ICSA certification programme - including developers, corporate end-users and independent researchers - are arguing that even 60 per cent efficiency contributes to reducing the spam problem.

In our own use of anti-spam packages like SpamPal and MailWasher, we found it fairly straightforward to achieve (we estimate) efficiency levels for blocking spam of 80 per cent or above. However this has at the expense of occasional (less than one in 100) false positives.

There is a disadvantage in using these products of mail downloads slowing to a crawl but the overall benefits of these packages outweigh their pitfalls. We also hear good things about SpamAssassin.

Doubtless other products and services achieve better results but without comprehensive independent testing it's hard to be sure what package to use.

The recent proliferation of anti-spam products and difficulties in agreeing testing criteria are complicating factors.. In testing only eight products, ICSA Labs is looking at only a very small sample of the scores of packages out there - a point that that hasn't gone unnoticed at ICSA.

We shouldn't read too much into ICSA Labs preliminary results, but they do point to the immaturity of the anti-spam market.

Longer term the future looks much brighter. With the best minds in the IETF, and elsewhere, focusing on the problem, gives reason for optimism.

Current research efforts to apply Bayesian analysis in weeding out spam messages from legitimate email and to slow spammers down using tar pits appear most promising. ®

Related stories

On spam cures that are worse than the disease
Web giants to declare war on spam
Florida spammers sue anti-spam groups
Rise of the Spam Zombies
IETF aims to can spam
Spammers break law with covert tracking
Plaid up in arms as Commons spam filter bans Welsh
Vendors complete tougher ICSA 4.0 firewall tests