Feeds

Search engines stink, and they're getting worse

An academic report finds them all sadly wanting

  • alert
  • submit to reddit

The essential guide to IT transformation

The proportion of information on the Internet that is indexed by search engines is declining, according to a recent study by Steve Lawrence and Lee Giles of the NEC Research Institute at Princeton, and reported in Nature. The engines do not index sites equally, new pages may remain unindexed for months, but worst of all, even the best engine only reaches 16 per cent of the Web. The survey was carried out in February. Furthermore, the situation is getting worse, since in December 1997 around 34 per cent of information was indexed. The problem is multifaceted. The Web has around 15 terabytes of data in some 800 million pages, plus 180 million images and is growing faster than the ability of the engines to search. The growth rate is about 3 million pages/day. There is apparently no coordination between search engine operators it seems, so that the cognoscenti could turn to a particular engine for a particular subject area. The dark side is of course that many pages that have made it to a search engine disappear without trace. So how well are the best-known engines doing? Dismally, is the answer. The best, according to Lawrence & Giles, is Northern Light, which covers a mere 16 per cent of the Web, just pipping Alta Vista's 15.5 percent (although that 0.5 per cent difference adds up to around 4 million unindexed pages). Microsoft can only manage 8.5 per cent, Yahoo 7.4 per cent, Excite 5.6 per cent, and Lycos is the dunce at 2.5 per cent. This should make people think about defaulting to using engines on portals. Of course enlightened searchers use meta-engines that use several engines and combine the results, but each has its inconveniences and idiosyncrasies we have found. The researchers found that 83 per cent of web sites have commercial content, with only a vociferous 1.5 per cent of sites being pornographic: they, at least, have found the secrets of tweaking their sites to get them indexed, it would appear. It looks as though archivists will not be out of a job for a long time, in view of this failure of the Web as a reliable and comprehensive online library. It reminds us of the persistent story that the French Bibliotheque Nationale used to store its books by size and colour in its old building. That's effectively what's happening on the Web: we don't know how much is unindexed, because it is hard to study the overlap between engines, but the odds are that half the information on the Web cannot be found with search engines at all. ®

Boost IT visibility and business value

More from The Register

next story
6 Obvious Reasons Why Facebook Will Ban This Article (Thank God)
Clampdown on clickbait ... and El Reg is OK with this
Banking apps: Handy, can grab all your money... and RIDDLED with coding flaws
Yep, that one place you'd hoped you wouldn't find 'em
No, thank you. I will not code for the Caliphate
Some assignments, even the Bongster decline must
Barnes & Noble: Swallow a Samsung Nook tablet, please ... pretty please
Novelslab finally on sale with ($199 - $20) price tag
Video of US journalist 'beheading' pulled from social media
Yanked footage featured British-accented attacker and US journo James Foley
Primetime precrime? Minority Report TV series 'being developed'
I have to know. I have to find out what happened to my life
Broadband slow and expensive? Blame Telstra says CloudFlare
Won't peer, will gouge for Internet transit
Netflix swallows yet another bitter pill, inks peering deal with TWC
Net neutrality crusader once again pays up for priority access
prev story

Whitepapers

Best practices for enterprise data
Discussing how technology providers have innovated in order to solve new challenges, creating a new framework for enterprise data.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Advanced data protection for your virtualized environments
Find a natural fit for optimizing protection for the often resource-constrained data protection process found in virtual environments.
How modern custom applications can spur business growth
Learn how to create, deploy and manage custom applications without consuming or expanding the need for scarce, expensive IT resources.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?