Feeds

GCHQ goes Google

Net spies turn to MapReduce

Build a business case: developing custom apps

Britain's digital spies have turned to Google for help making sense of the floods of data now inundating their powerful computing resources.

GCHQ, the Cheltenham-based signals intelligence agency, is recruiting an expert on MapReduce, the patented number-crunching technique previously behind the dominant web search engine.

The agency's new lead researcher on data mining will be responsible for "developing MapReduce analytics on parallel computing clusters", a job advertisment reveals.

MapReduce was developed by Google to index billions of web pages across its cluster of hundreds of thousands of commodity servers. It breaks up complicated tasks into smaller, easier computing problems that cheap hardware is capable of solving quickly.

Google patented the technique earlier this year, but it remains free for other organisations to adopt via Hadoop, an open source project. Originally described in a 2004 research paper, MapReduce has allowed Google's algorithms to index a rapidly expanding web while keeping costs down.

GCHQ faces similar a challenge as it gathers more and more raw data from internet communications, including email, social networks and VoIP.

"Successful data-driven organisations must be able to process, interpret and rapidly respond to indicators derived from unprecedented volumes of data from disparate information sources," its recruitment advertisement says.

The Register understands that GCHQ now has a cluster of more than 250,000 commodity servers under its Cheltenham "doughnut" building. In recent years it has developed this Google-style infrastructure instead of the very expensive, bespoke supercomputers it used to analyse microwave intercepts during the Cold War.

While spies are planning research on MapReduce, Google has already moved on to BigTable, its new distributed database. ®

Build a business case: developing custom apps

More from The Register

next story
Just TWO climate committee MPs contradict IPCC: The two with SCIENCE degrees
'Greenhouse effect is real, but as for the rest of it ...'
Adam Afriyie MP: Smart meters are NOT so smart
Mega-costly gas 'n' 'leccy totting-up tech not worth it - Tory MP
'Blow it up': Plods pop round for chat with Commonwealth Games tweeter
You'd better not be talking about the council's housing plans
Arrr: Freetard-bothering Digital Economy Act tied up, thrown in the hold
Ministry of Fun confirms: Yes, we're busy doing nothing
ONE EMAIL costs mining company $300 MEEELION
Environmental activist walks free after hoax sent share price over a cliff
Help yourself to anyone's photos FOR FREE, suggests UK.gov
Copyright law reforms will keep m'learned friends busy
Apple smacked with privacy sueball over Location Services
Class action launched on behalf of 100 million iPhone owners
UK government officially adopts Open Document Format
Microsoft insurgency fails, earns snarky remark from UK digital services head
prev story

Whitepapers

Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
The Essential Guide to IT Transformation
ServiceNow discusses three IT transformations that can help CIO's automate IT services to transform IT and the enterprise.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
How modern custom applications can spur business growth
Learn how to create, deploy and manage custom applications without consuming or expanding the need for scarce, expensive IT resources.
Build a business case: developing custom apps
Learn how to maximize the value of custom applications by accelerating and simplifying their development.