Feeds

Search engines stink, and they're getting worse

An academic report finds them all sadly wanting

  • alert
  • submit to reddit

Internet Security Threat Report 2014

The proportion of information on the Internet that is indexed by search engines is declining, according to a recent study by Steve Lawrence and Lee Giles of the NEC Research Institute at Princeton, and reported in Nature. The engines do not index sites equally, new pages may remain unindexed for months, but worst of all, even the best engine only reaches 16 per cent of the Web. The survey was carried out in February. Furthermore, the situation is getting worse, since in December 1997 around 34 per cent of information was indexed. The problem is multifaceted. The Web has around 15 terabytes of data in some 800 million pages, plus 180 million images and is growing faster than the ability of the engines to search. The growth rate is about 3 million pages/day. There is apparently no coordination between search engine operators it seems, so that the cognoscenti could turn to a particular engine for a particular subject area. The dark side is of course that many pages that have made it to a search engine disappear without trace. So how well are the best-known engines doing? Dismally, is the answer. The best, according to Lawrence & Giles, is Northern Light, which covers a mere 16 per cent of the Web, just pipping Alta Vista's 15.5 percent (although that 0.5 per cent difference adds up to around 4 million unindexed pages). Microsoft can only manage 8.5 per cent, Yahoo 7.4 per cent, Excite 5.6 per cent, and Lycos is the dunce at 2.5 per cent. This should make people think about defaulting to using engines on portals. Of course enlightened searchers use meta-engines that use several engines and combine the results, but each has its inconveniences and idiosyncrasies we have found. The researchers found that 83 per cent of web sites have commercial content, with only a vociferous 1.5 per cent of sites being pornographic: they, at least, have found the secrets of tweaking their sites to get them indexed, it would appear. It looks as though archivists will not be out of a job for a long time, in view of this failure of the Web as a reliable and comprehensive online library. It reminds us of the persistent story that the French Bibliotheque Nationale used to store its books by size and colour in its old building. That's effectively what's happening on the Web: we don't know how much is unindexed, because it is hard to study the overlap between engines, but the odds are that half the information on the Web cannot be found with search engines at all. ®

Top 5 reasons to deploy VMware with Tegile

More from The Register

next story
Facebook pays INFINITELY MORE UK corp tax than in 2012
Thanks for the £3k, Zuck. Doh! you're IN CREDIT. Guess not
Facebook, Apple: LADIES! Why not FREEZE your EGGS? It's on the company!
No biological clockwatching when you work in Silicon Valley
Happiness economics is bollocks. Oh, UK.gov just adopted it? Er ...
Opportunity doesn't knock; it costs us instead
Sysadmin with EBOLA? Gartner's issued advice to debug your biz
Start hoarding cleaning supplies, analyst firm says, and assume your team will scatter
Edward who? GCHQ boss dodges Snowden topic during last speech
UK spies would rather 'walk' than do 'mass surveillance'
YARR! Pirates walk the plank: DMCA magnets sink in Google results
Spaffing copyrighted stuff over the web? No search ranking for you
Don't bother telling people if you lose their data, say Euro bods
You read that right – with the proviso that it's encrypted
prev story

Whitepapers

Cloud and hybrid-cloud data protection for VMware
Learn how quick and easy it is to configure backups and perform restores for VMware environments.
A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Three 1TB solid state scorchers up for grabs
Big SSDs can be expensive but think big and think free because you could be the lucky winner of one of three 1TB Samsung SSD 840 EVO drives that we’re giving away worth over £300 apiece.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.