Feeds

Internet could be 500 times bigger than we think

And very easy to kick over

  • alert
  • submit to reddit

The essential guide to IT transformation

The true extent of the Internet is not widely known, and according to a study published this week, it could be more than 500 times larger than we think. The authors claim that as much as 7500TB of data exist in places on the Web that no search engine has mapped, as compared to the 19TB on the familiar "surface" Web.

The material BrightPlanet has uncovered consists mostly of public information - 95 per cent of this is freely accessible, with more than half residing in topic-specific databases. Of these, the 60 largest contain 750TB of information. This exceeds the capacity of the "normal" Web by 40 times.

All this hidden material is causing a great deal of frustration, the company says, as people can't find it using the usual search engines. Even NorthernLight.com, widely reported to have the largest percentage of the web mapped at 16 per cent, covers only 0.03 per cent of total content including the "deep" Web.

Fortunately, and perhaps unsurprisingly, BrightPlanet has the solution in the form of its own new software called Lexibot. This will search the surface Web, and will access the online databases to search for information there as well.

However, this does not happen fast. In fact shifting continents can just about keep up. It will take on average 10 to 25 minutes to fill a search request, the executives at the company estimate. But complex queries could take anything up to an hour and a half. And uniquely, as far as we know, the software also costs money. More specifically, it costs $89.95 following a free 30-day trial.

In the same week, New Scientist reports that the Internet is not just bigger than we think, but also more vulnerable to sabotage than we imagine.

According to a mathematical model published this week, if the right nodes were targeted, the network would quickly break down into isolated pieces and stop working. However, the Internet could withstand 18 per cent of its nodes being taken down randomly.

The study was investigating the differences between an exponential network and one that is scale-free, like the Internet. The exponential network under random attack loses performance quickly since all the nodes are equally important. A scale-free network is far more robust in this respect. But when it comes to targeted attacks, the exponential network has no obvious weak point to shoot for, and handles such an assault far better.

Researchers say that this means network operators can focus resources to provide security for the really vital parts of the Internet's network such as the backbone systems that carry most of the traffic.®

Related links

The search for the perfect search engine

Gartner critical capabilities for enterprise endpoint backup

More from The Register

next story
6 Obvious Reasons Why Facebook Will Ban This Article (Thank God)
Clampdown on clickbait ... and El Reg is OK with this
No, thank you. I will not code for the Caliphate
Some assignments, even the Bongster decline must
Barnes & Noble: Swallow a Samsung Nook tablet, please ... pretty please
Novelslab finally on sale with ($199 - $20) price tag
Banking apps: Handy, can grab all your money... and RIDDLED with coding flaws
Yep, that one place you'd hoped you wouldn't find 'em
Video of US journalist 'beheading' pulled from social media
Yanked footage featured British-accented attacker and US journo James Foley
Primetime precrime? Minority Report TV series 'being developed'
I have to know. I have to find out what happened to my life
Netflix swallows yet another bitter pill, inks peering deal with TWC
Net neutrality crusader once again pays up for priority access
Judge nixes HP deal for director amnesty after $8.8bn Autonomy snafu
Lawyers will have to earn their keep the hard way, says court
Ex-IBM CEO John Akers dies at 79
An era disrupted by the advent of the PC
prev story

Whitepapers

Top 10 endpoint backup mistakes
Avoid the ten endpoint backup mistakes to ensure that your critical corporate data is protected and end user productivity is improved.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Backing up distributed data
Eliminating the redundant use of bandwidth and storage capacity and application consolidation in the modern data center.
The essential guide to IT transformation
ServiceNow discusses three IT transformations that can help CIOs automate IT services to transform IT and the enterprise
Next gen security for virtualised datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.