How to beat AVG's fake traffic spew
A header to save the internet
Updated As the AVG LinkScanner continues to spew fake traffic across the internet, web masters say they've uncovered a reliable means of filtering these rogue hits from their log files.
Bundled with AVG's newest anti-virus engine, AVG 8, and used by roughly 20 million people worldwide, LinkScanner checks search engine results for malware before you on click them. If you type a keyword into Google, for instance, it automatically visits each address that appears on Google's results page.
This has caused an enormous spike in traffic on sites across the web, including The Register, and many webmasters may not realize where these hits are coming from. Hoping to fool malware writers, LinkScanner mimics real live human clicks. At least in part.
When scanning pages, LinkScanner employs the IP addresses of those 20 million people who use the product, and as of last week, it sends the same user agent as Microsoft's IE6 browser.
But there is a way of eliminating this fake traffic from log files - without clipping the real thing. Two separate webmasters have learned - through two different avenues - that an AVG request leaves out at least one HTTP header: the "Accept-Encoding" header, which defines what sort of compression a browser can handle.
Filtering or redirecting traffic according to this header (and the IE6 user agent) should do the trick - though AVG did not respond when asked for confirmation.
AVG has said "There are still ways for concerned web masters to filter LinkScanner requests out of their statistics." But it hasn't given specifics - at least publicly. And any fix may be temporary.
Part of the problem is that AVG won't put its cards on the table. But the company may worry that divulging too much information would feed the black hats. If a web master can identify AVG scans, so can a malware writer. But more on that later.
Meanwhile, web masters continue to complain that LinkScanner forces them to pay for extra bandwidth. But AVG has promised a fix for this problem as well. ®
Many Reg readers have asked why LinkScanner doesn't scans links after you click on them. But it does. It scans both before and after. AVG chief of research Roger Thompson argues that this two-layer approach guards against so-called zero-day exploits. If a site is infected with one piece of malware, there's a good chance it's infected with a second, he says, and if you prevent users from even clicking on a site, you protect them from exploits you see as well as those you don't.
To avoid confusion, we've updated this story to point out that webmasters are filtering traffic by identifying both the missing HTTP header and the IE6 agent. It should also be noted that LinkScanner seems to be using two other user agents. But these are far less prevalent. At the moment, these are the three user agents the product seems to employ:
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1;1813)
The first is the IE6 agent.
Does anyone know if it visits the sponsored pay per click links as well the natural/organic results please?
That could prove to be quite expensive for the advertisers if every search where their ad gets an impression is getting a clickthrough.
AVG Responds to and Resolves LinkScanner Issues
AVG has already responded to resolve this issue. The full response can be seen at http://www.avg.com.au/index.cfm?section=news&feature=104
An updated version of AVG Anti-Virus Free Edition 8.0 is already available, see http://www.avgfree.com.au. The Search-Shield component of LinkScanner has been modified to only notify users of malicious sites. The equivalent modification to the the AVG 8.0 commercial products will be rolled out on 9th July 2008.
Once the updated version has been rolled out to all AVG 8.0 users the issue will be resolved.
As of this date, Search-Shield will no longer scan each search result online for new exploits, which was causing the spikes that web masters addressed with us.
However, it is important to note that AVG still offers full protection against potential exploits through the LinkScanner Active Surf-Shield component of our product, which checks every page for malicious content as it is visited but before it is opened.
We’d like to thank the web community for bringing these challenges to our attention, as building community trust and protecting all of our users is critical to us.
Best Regards, Lloyd Borrett
Marketing Manager, AVG (AU/NZ)
You guys might be interested in this. We just got a response from the CEO of AVG Australia.
"It's Peter Cameron here. I am the Managing Director of AVG Australia / NZ and I thought that is was important to communicate with users of Whirlpool tonight. I have spent the last two hours reading your posts and have heard your concerns loud and clear.
I can categorically assure every member here that AVG has heard you, and that we fully understand the issues that have been raised here and that this message has been loudly communicated to the AVG Technologies development teams in both the USA and Europe. I assure you that you currently have the attention of the Chief Technology Officer who is the person responsible for the design and development of the AVG product range.
As a matter of urgency, AVG is evaluating the LinkScanner technology so that we can ensure we maintain the level of protection for AVG users, but to minimise the impact on web sites and minimise any bandwidth overheads to both AVG users and web hosts.
We expect the AVG development team will respond quite soon with an effective resolution to the concerns raised both here on Whirlpool and elsewhere.
AVG has always worked to provide maximum protection for our users and also for the Internet eco-system (it is one of the reasons behind the AVG Anti-Virus Free Edition product) and we will continue to strive towards these goals.
AVG Australia / New Zealand"
Has exploit hit AVG?
Yesterday ( July 2 ) noticed my AVG8 had last updated and scanned on June 28. Tried manually updating with questionable results, could not reset automatic updates, scan took f o r e v e r, went to AVG site, couldn't successfully register to post in Free forum so had to be satisfied with reading. Seems many users there ( beginning around June 26 or so ) were having same difficulties. Finally found info that fix was to download newest installer ( released July 2 ) and choose "Repair" but off-site link was older installer and on-site link was miserably slow ( 2-5kbs ) - gritted teeth and said I'd give 'em one more day - same results today. Removed AVG & installed Avira AntiVir Personal. Too much grief. I think someone targeted AVG8 and they were too slow in coming up with a fix and too limited in making it available to those of us who wanted it. The Register should dig in and find out what happened to those of us who put our trust in AVG but won't again anytime soon. Betcha AVG won't let on - the moderators in the forum didn't, the most revealing thing there were "servers are overloaded" comments, which was definitely true, but no info as to why. I think their boat has been sunk.
Y'all are still missing the boat
@AC recommending avg.load("/js/empty.js")
Again, you CAN NOT rely on people having JS enabled, cookies enabled or any other option enabled. More and more people are becoming security conscious, or having a security conscious friend drill this stuff into their thick head. I personally run with a *.js block rule at the proxy. I don't need no stinking scripts.
LinkScanner offers absolutely no technical advantage. Why would they have a better chance of spotting malware on a preemtive scan than on the actual download to the browser? They wouldn't! This is nothing but a marketing ploy to establish a feature where they are "ahead" of the pack. Its straight up FUD, plain and simple.
Paris. Because only she would be dumb enough to still be finding "solutions" to this.