Feeds

GCHQ goes Google

Net spies turn to MapReduce

The essential guide to IT transformation

Britain's digital spies have turned to Google for help making sense of the floods of data now inundating their powerful computing resources.

GCHQ, the Cheltenham-based signals intelligence agency, is recruiting an expert on MapReduce, the patented number-crunching technique previously behind the dominant web search engine.

The agency's new lead researcher on data mining will be responsible for "developing MapReduce analytics on parallel computing clusters", a job advertisment reveals.

MapReduce was developed by Google to index billions of web pages across its cluster of hundreds of thousands of commodity servers. It breaks up complicated tasks into smaller, easier computing problems that cheap hardware is capable of solving quickly.

Google patented the technique earlier this year, but it remains free for other organisations to adopt via Hadoop, an open source project. Originally described in a 2004 research paper, MapReduce has allowed Google's algorithms to index a rapidly expanding web while keeping costs down.

GCHQ faces similar a challenge as it gathers more and more raw data from internet communications, including email, social networks and VoIP.

"Successful data-driven organisations must be able to process, interpret and rapidly respond to indicators derived from unprecedented volumes of data from disparate information sources," its recruitment advertisement says.

The Register understands that GCHQ now has a cluster of more than 250,000 commodity servers under its Cheltenham "doughnut" building. In recent years it has developed this Google-style infrastructure instead of the very expensive, bespoke supercomputers it used to analyse microwave intercepts during the Cold War.

While spies are planning research on MapReduce, Google has already moved on to BigTable, its new distributed database. ®

5 things you didn’t know about cloud backup

More from The Register

next story
Munich considers dumping Linux for ... GULP ... Windows!
Give a penguinista a hug, the Outlook's not good for open source's poster child
UK fuzz want PINCODES on ALL mobile phones
Met Police calls for mandatory passwords on all new mobes
e-Borders fiasco: Brits stung for £224m after US IT giant sues UK govt
Defeat to Raytheon branded 'catastrophic result'
EU justice chief blasts Google on 'right to be forgotten'
Don't pretend it's a freedom of speech issue – interim commish
Yes, but what are your plans if a DRAGON attacks?
Local UK gov outs most ridiculous FoI requests...
Detroit losing MILLIONS because it buys CHEAP BATTERIES – report
Man at hardware store was right: name brands DO last longer
Snowden on NSA's MonsterMind TERROR: It may trigger cyberwar
Plus: Syria's internet going down? That was a US cock-up
UK government accused of hiding TRUTH about Universal Credit fiasco
'Reset rating keeps secrets on one-dole-to-rule-them-all plan', say MPs
Caught red-handed: UK cops, PCSOs, specials behaving badly… on social media
No Mr Fuzz, don't ask a crime victim to be your pal on Facebook
prev story

Whitepapers

5 things you didn’t know about cloud backup
IT departments are embracing cloud backup, but there’s a lot you need to know before choosing a service provider. Learn all the critical things you need to know.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Build a business case: developing custom apps
Learn how to maximize the value of custom applications by accelerating and simplifying their development.
Rethinking backup and recovery in the modern data center
Combining intelligence, operational analytics, and automation to enable efficient, data-driven IT organizations using the HP ABR approach.
Next gen security for virtualised datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.