Feeds

GCHQ goes Google

Net spies turn to MapReduce

5 things you didn’t know about cloud backup

Britain's digital spies have turned to Google for help making sense of the floods of data now inundating their powerful computing resources.

GCHQ, the Cheltenham-based signals intelligence agency, is recruiting an expert on MapReduce, the patented number-crunching technique previously behind the dominant web search engine.

The agency's new lead researcher on data mining will be responsible for "developing MapReduce analytics on parallel computing clusters", a job advertisment reveals.

MapReduce was developed by Google to index billions of web pages across its cluster of hundreds of thousands of commodity servers. It breaks up complicated tasks into smaller, easier computing problems that cheap hardware is capable of solving quickly.

Google patented the technique earlier this year, but it remains free for other organisations to adopt via Hadoop, an open source project. Originally described in a 2004 research paper, MapReduce has allowed Google's algorithms to index a rapidly expanding web while keeping costs down.

GCHQ faces similar a challenge as it gathers more and more raw data from internet communications, including email, social networks and VoIP.

"Successful data-driven organisations must be able to process, interpret and rapidly respond to indicators derived from unprecedented volumes of data from disparate information sources," its recruitment advertisement says.

The Register understands that GCHQ now has a cluster of more than 250,000 commodity servers under its Cheltenham "doughnut" building. In recent years it has developed this Google-style infrastructure instead of the very expensive, bespoke supercomputers it used to analyse microwave intercepts during the Cold War.

While spies are planning research on MapReduce, Google has already moved on to BigTable, its new distributed database. ®

The essential guide to IT transformation

More from The Register

next story
Britain's housing crisis: What are we going to do about it?
Rent control: Better than bombs at destroying housing
Top beak: UK privacy law may be reconsidered because of social media
Rise of Twitter etc creates 'enormous challenges'
GCHQ protesters stick it to British spooks ... by drinking urine
Activists told NOT to snap pics of staff at the concrete doughnut
What do you mean, I have to POST a PHYSICAL CHEQUE to get my gun licence?
Stop bitching about firearms fees - we need computerisation
Ex US cybersecurity czar guilty in child sex abuse website case
Health and Human Services IT security chief headed online to share vile images
We need less U.S. in our WWW – Euro digital chief Steelie Neelie
EC moves to shift status quo at Internet Governance Forum
Oz biz regulator discovers shared servers in EPIC FACEPALM
'Not aware' that one IP can hold more than one Website
prev story

Whitepapers

Endpoint data privacy in the cloud is easier than you think
Innovations in encryption and storage resolve issues of data privacy and key requirements for companies to look for in a solution.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Advanced data protection for your virtualized environments
Find a natural fit for optimizing protection for the often resource-constrained data protection process found in virtual environments.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Next gen security for virtualised datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.