Pass the Lizard-proof tinfoil, I need to make a hat

  • alert
  • submit to reddit

Boost IT visibility and business value

A story that the Conservatives “made the internet disappear” has ignited news channels today. In fact, the story demonstrates yet again how ignorant most journalists are of the basic workings of the internet - and it demonstrates how the thirst for conspiratorial thinking dominates political news.

Perhaps not surprisingly, this story comes from freelance blogger Mark Ballard, writing for twilight IT site Computer Weekly. Mark used to be a journalist here at The Register: he had to leave some years ago, but we remember him well.

Ballard reports in classic style that the Conservative Party has "erased" a bunch of old speeches from the "internet". Sky News, picking up Ballard's post, tells us how “the Tories had used something called a robot blocker to remove speeches”, echoing the crazed conspiratorial tone of the Weakly's coverage - which adds:

The erasure had the effect of hiding Conservative speeches in a secretive corner of the internet like those that shelter the military, secret services, gangsters and paedophiles.

In fact, it's simpler than that: the Tories just dumped all their old speeches off their website and put notices in their robots.txt file notifying webcrawlers ("the robots" that the Tories "blocked", if you've wandered in here from somewhere else) to do the same in their parent archives around the internet**. Mark and other scribblers at the Beeb, Sky, Guardian, BuzzFeed etc were stunned to find that this meant the speeches were no longer available at the Wayback Machine, which they had fondly imagined to be "the internet", and to serve as an imperishable archive of everything ever published online.

But this is business-as-usual. The shock and horror is generated by a misconception: that the internet in general, and the Wayback Machine (aka the library hosted by non-profit firm Archive.org) in particular, are bulletproof repositories of information. The word "archive" implies permanence – a fortress of data integrity impervious to time, war and large egos. But Archive.org is not an archive. It never has been.

We found this out nine years ago. The Wayback Machine is very fragmentary, and - of course - removes information on request. On that occasion the PR department of chip giant Intel had requested that a three-year-old interview with an engineer be removed. Wayback complied – as it does every time. After all, it seldom has any right to copy and republish content scraped from other people's websites.

There are other reasons why it's no surprise that the Wayback crawlers comply with robots.txt files on websites: if a crawler doesn't, it is liable to be blocked by all right-thinking webmasters. Dubious bot crawlers often try to pretend not to be robots, so that they can ignore robots.txt, but this isn't simple - they are often detected by alert web admins or their systems - and that's a dangerous route to go down when you're claiming to be a reputable organisation.

So there are good bots and bad bots, on the real internet. But not on Mark Ballard's. He writes:

The bots were what made the democratization of information possible. It was bots that inspired Cameron and Osborne. It was bots that were going to free us from serfdom in the way they said we would be. Without the bots you just had pockets of power and privilege for those in the know. Without the bots you just had the same old concentration of wealth and power there had always been, since long before the Internet Archive started taking snapshots of the Conservative website in 1999.

Knockabout stuff. And the Tories taking all their old promises off their website - and updating their robots.txt to reflect this - would almost be a small piece of news on a slow day, though all the "erasing the internet" and "criminals and paedophiles" foolery is absurd.

Except it would only be fair to note that the Labour Party has "erased the internet" too. Labour’s housecleaning has removed almost everything prior to the start of the current leadership, and the Wayback Machine (sorry, "the internet") is pretty empty of Labour's past as well as that of the Tories.

So, not really even news: "political parties scrub away all their old promises as long run-up to election begins". Boring.

If we’re to hold politicians to account then this means proper, rational debate. That means, yes, keeping a record of what they say - but you're better off taking your own copies than relying on robots and the varied cloud systems they serve to do it for you - and then complaining that someone has "erased the internet" like a "criminal paedophile" when you are let down.

But people would normally much prefer a conspiracy to blame. Once it was the Right that dealt largely in the language conspiracy theories, as Democrat historian Richard Hofstader wrote in his famous 1964 essay The Paranoid Style in American Politics. There were Reds to be found under every Bed. But listen to any academic venting about “neoliberalism” and you can just as easily erase the word “neoliberal” and substitute the word “Illuminati”. It's escapism and a form of narcissism, really.

Every conspiracy theorist finds the conspiracy their heart desires, eventually. Readers may be interested to note that when Mark left the Reg, he took with him several folders of notes, and left behind quite a lot more.

The ones he took with him were all labelled "Military Industrial Complex", followed by sequential serial numbers. ®

*Headlines to which the answer is no.

** Newcomers might care to have a look here to learn more about the robot exclusion protocol, which "is not intended for access control, so don't try to use it as such. Think of it as a 'No Entry' sign, not a locked door."

Build a business case: developing custom apps

More from The Register

next story
Detroit losing MILLIONS because it buys CHEAP BATTERIES – report
Man at hardware store was right: name brands DO last longer
Yes, but what are your plans if a DRAGON attacks?
Local UK gov outs most ridiculous FoI requests...
Munich considers dumping Linux for ... GULP ... Windows!
Give a penguinista a hug, the Outlook's not good for open source's poster child
Govt control? Hah! It's IMPOSSIBLE to have a successful command economy
Even Moore's Law can't help the architects of statism now
Snowden on NSA's MonsterMind TERROR: It may trigger cyberwar
Plus: Syria's internet going down? That was a US cock-up
New voting rules leave innocent Brits at risk of SPAM TSUNAMI
Read the paperwork very carefully - or fall victim to marketing shysters
e-Borders fiasco: Brits stung for £224m after US IT giant sues UK govt
Defeat to Raytheon branded 'catastrophic result'
prev story


Endpoint data privacy in the cloud is easier than you think
Innovations in encryption and storage resolve issues of data privacy and key requirements for companies to look for in a solution.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Top 8 considerations to enable and simplify mobility
In this whitepaper learn how to successfully add mobile capabilities simply and cost effectively.
Solving today's distributed Big Data backup challenges
Enable IT efficiency and allow a firm to access and reuse corporate information for competitive advantage, ultimately changing business outcomes.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.