Feeds

Internet could be 500 times bigger than we think

And very easy to kick over

  • alert
  • submit to reddit

Intelligent flash storage arrays

The true extent of the Internet is not widely known, and according to a study published this week, it could be more than 500 times larger than we think. The authors claim that as much as 7500TB of data exist in places on the Web that no search engine has mapped, as compared to the 19TB on the familiar "surface" Web.

The material BrightPlanet has uncovered consists mostly of public information - 95 per cent of this is freely accessible, with more than half residing in topic-specific databases. Of these, the 60 largest contain 750TB of information. This exceeds the capacity of the "normal" Web by 40 times.

All this hidden material is causing a great deal of frustration, the company says, as people can't find it using the usual search engines. Even NorthernLight.com, widely reported to have the largest percentage of the web mapped at 16 per cent, covers only 0.03 per cent of total content including the "deep" Web.

Fortunately, and perhaps unsurprisingly, BrightPlanet has the solution in the form of its own new software called Lexibot. This will search the surface Web, and will access the online databases to search for information there as well.

However, this does not happen fast. In fact shifting continents can just about keep up. It will take on average 10 to 25 minutes to fill a search request, the executives at the company estimate. But complex queries could take anything up to an hour and a half. And uniquely, as far as we know, the software also costs money. More specifically, it costs $89.95 following a free 30-day trial.

In the same week, New Scientist reports that the Internet is not just bigger than we think, but also more vulnerable to sabotage than we imagine.

According to a mathematical model published this week, if the right nodes were targeted, the network would quickly break down into isolated pieces and stop working. However, the Internet could withstand 18 per cent of its nodes being taken down randomly.

The study was investigating the differences between an exponential network and one that is scale-free, like the Internet. The exponential network under random attack loses performance quickly since all the nodes are equally important. A scale-free network is far more robust in this respect. But when it comes to targeted attacks, the exponential network has no obvious weak point to shoot for, and handles such an assault far better.

Researchers say that this means network operators can focus resources to provide security for the really vital parts of the Internet's network such as the backbone systems that carry most of the traffic.®

Related links

The search for the perfect search engine

Internet Security Threat Report 2014

More from The Register

next story
The 'fun-nification' of computer education – good idea?
Compulsory code schools, luvvies love it, but what about Maths and Physics?
Facebook, Apple: LADIES! Why not FREEZE your EGGS? It's on the company!
No biological clockwatching when you work in Silicon Valley
Lords take revenge on REVENGE PORN publishers
Jilted Johns and Jennies with busy fingers face two years inside
Happiness economics is bollocks. Oh, UK.gov just adopted it? Er ...
Opportunity doesn't knock; it costs us instead
Ex-US Navy fighter pilot MIT prof: Drones beat humans - I should know
'Missy' Cummings on UAVs, smartcars and dying from boredom
Yes, yes, Steve Jobs. Look what I'VE done for you lately – Tim Cook
New iPhone biz baron points to Apple's (his) greatest successes
Sysadmin with EBOLA? Gartner's issued advice to debug your biz
Start hoarding cleaning supplies, analyst firm says, and assume your team will scatter
Edward who? GCHQ boss dodges Snowden topic during last speech
UK spies would rather 'walk' than do 'mass surveillance'
Doctor Who's Flatline: Cool monsters, yes, but utterly limp subplots
We know what the Doctor does, stop going on about it already
prev story

Whitepapers

Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Why and how to choose the right cloud vendor
The benefits of cloud-based storage in your processes. Eliminate onsite, disk-based backup and archiving in favor of cloud-based data protection.
Three 1TB solid state scorchers up for grabs
Big SSDs can be expensive but think big and think free because you could be the lucky winner of one of three 1TB Samsung SSD 840 EVO drives that we’re giving away worth over £300 apiece.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.