Feeds

My spam-filled search index is bigger than yours!

NCSA study backfires

Top three mobile application threats

Last week Yahoo! claimed it had sailed past Google by indexing 20 billion web pages.

Because as much as a third of the wild wild web consists of artificially-generated pages of spam designed to promote commercial web sites, this isn't much to boast about. Many of the fake pages are 'splogs', or spam blogs, or phoney catalogs, or simply pages of dictionary words. You can meet one of the perps here, in our story Interview with a link spammer.

And because few users have the patience to find the gem returned at position #12,711 in the search results, then the size of an index is meaningless.

More is never a substitute for better.

Unofficial Google spokesperson John Battelle has disputed the claim, and now the supercomputer center NCSA has poured cold water on the boast, too.

In a sample comparison, three researchers University of Illinois at Urbana-Champaign found that Google returned more search results than Yahoo! They conclude,

"It is the opinion of this study that Yahoo!'s claim to have a web index of over twice as many documents as Google's index is suspicious. Unless a large number of the documents Yahoo! has indexed are not yet available to its search engine, we find it puzzling that Yahoo!'s search engine consistently returned fewer results than Google."

A bold claim, for sure. There's a problem, however - NCSA's results are a little too incredible.

As Seth Finkelstein points out on his weblog, the NCSA researchers fed random dictionary words into both search engines - and not surprisingly, the results reflect pages containing random dictionary words.

"By sampling random words, they biased the samples to files of large word lists! And this effect applies, to a great or lesser extent, to every sample.

Here's an example. For the spam friendly gibberish words "carbolization clambers" Google returned 7 pages, all from a dictionary, and Yahoo! returned none. For the words " alkaloid's observance", Google returns 30 pages and Yahoo none.

In other words, the methodology is geared not to measure who has the most useful documents, but who has the most spam. To be more precise, in these examples, Google returns a number of copies of a dictionary file. It's a different frequency of noise.

"Just to show the problem, imagine that Google had returned results of three dictionary files, and Yahoo one dictionary file," explains Finkelstein. "Do this *10,000* times, and you get Google returning 30,000 results, and Yahoo! returning 10,000 results. So, wow, Google has 3x the size! But, in fact, it's just the same little quirk being counted 10,000 times over!"

The intrepid NCSA researchers appear to be blissfully unaware of that they've marched into a swamp.

"We feel that small, randomly selected search queries gives us the best chance to locate some of the most obscure web documents," they insist. Nevertheless, they plough bravely on.

"By counting the presence of these obscure documents in either search engine, we can measure the comprehensiveness of each search engine to determine the relative size of each search engine's index," they maintain.

Oh, dear.

The Argentian writer Borges described the Falklands War as "two bald men fighting over a comb", and it's an apt description for the spectacle of the search engines, and their rival supporters, duking it out to boast who's has collected the most garbage. ®

The Essential Guide to IT Transformation

More from The Register

next story
BBC goes offline in MASSIVE COCKUP: Stephen Fry partly muzzled
Auntie tight-lipped as major outage rolls on
iPad? More like iFAD: We reveal why Apple ran off to IBM
But never fear fanbois, you're still lapping up iPhones, Macs
Nadella: Apps must run on ALL WINDOWS – PCs, slabs and mobes
Phone egg, meet desktop chicken - your mother
ITC: Seagate and LSI can infringe Realtek patents because Realtek isn't in the US
Land of the (get off scot) free, when it's a foreign owner
HP, Microsoft prove it again: Big Business doesn't create jobs
SMEs get lip service - what they need is dinner at the Club
Samsung threatens to cut ties with supplier over child labour allegations
Vows to uphold 'zero tolerance' policy on underage workers
Dude, you're getting a Dell – with BITCOIN: IT giant slurps cryptocash
1. Buy PC with Bitcoin. 2. Mine more coins. 3. Goto step 1
There's NOTHING on TV in Europe – American video DOMINATES
Even France's mega subsidies don't stop US content onslaught
You! Pirate! Stop pirating, or we shall admonish you politely. Repeatedly, if necessary
And we shall go about telling people you smell. No, not really
prev story

Whitepapers

Seven Steps to Software Security
Seven practical steps you can begin to take today to secure your applications and prevent the damages a successful cyber-attack can cause.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Designing a Defense for Mobile Applications
Learn about the various considerations for defending mobile applications - from the application architecture itself to the myriad testing technologies.
Build a business case: developing custom apps
Learn how to maximize the value of custom applications by accelerating and simplifying their development.
Consolidation: the foundation for IT and business transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.