Feeds

GCHQ goes Google

Net spies turn to MapReduce

Mobile application security vulnerability report

Britain's digital spies have turned to Google for help making sense of the floods of data now inundating their powerful computing resources.

GCHQ, the Cheltenham-based signals intelligence agency, is recruiting an expert on MapReduce, the patented number-crunching technique previously behind the dominant web search engine.

The agency's new lead researcher on data mining will be responsible for "developing MapReduce analytics on parallel computing clusters", a job advertisment reveals.

MapReduce was developed by Google to index billions of web pages across its cluster of hundreds of thousands of commodity servers. It breaks up complicated tasks into smaller, easier computing problems that cheap hardware is capable of solving quickly.

Google patented the technique earlier this year, but it remains free for other organisations to adopt via Hadoop, an open source project. Originally described in a 2004 research paper, MapReduce has allowed Google's algorithms to index a rapidly expanding web while keeping costs down.

GCHQ faces similar a challenge as it gathers more and more raw data from internet communications, including email, social networks and VoIP.

"Successful data-driven organisations must be able to process, interpret and rapidly respond to indicators derived from unprecedented volumes of data from disparate information sources," its recruitment advertisement says.

The Register understands that GCHQ now has a cluster of more than 250,000 commodity servers under its Cheltenham "doughnut" building. In recent years it has developed this Google-style infrastructure instead of the very expensive, bespoke supercomputers it used to analyse microwave intercepts during the Cold War.

While spies are planning research on MapReduce, Google has already moved on to BigTable, its new distributed database. ®

Bridging the IT gap between rising business demands and ageing tools

More from The Register

next story
Adam Afriyie MP: Smart meters are NOT so smart
Mega-costly gas 'n' 'leccy totting-up tech not worth it - Tory MP
'Blow it up': Plods pop round for chat with Commonwealth Games tweeter
You'd better not be talking about the council's housing plans
Arrr: Freetard-bothering Digital Economy Act tied up, thrown in the hold
Ministry of Fun confirms: Yes, we're busy doing nothing
ONE EMAIL costs mining company $300 MEEELION
Environmental activist walks free after hoax sent share price over a cliff
Help yourself to anyone's photos FOR FREE, suggests UK.gov
Copyright law reforms will keep m'learned friends busy
Apple smacked with privacy sueball over Location Services
Class action launched on behalf of 100 million iPhone owners
UK government officially adopts Open Document Format
Microsoft insurgency fails, earns snarky remark from UK digital services head
You! Pirate! Stop pirating, or we shall admonish you politely. Repeatedly, if necessary
And we shall go about telling people you smell. No, not really
prev story

Whitepapers

Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Application security programs and practises
Follow a few strategies and your organization can gain the full benefits of open source and the cloud without compromising the security of your applications.
How modern custom applications can spur business growth
Learn how to create, deploy and manage custom applications without consuming or expanding the need for scarce, expensive IT resources.
Securing Web Applications Made Simple and Scalable
Learn how automated security testing can provide a simple and scalable way to protect your web applications.