Feeds

GCHQ goes Google

Net spies turn to MapReduce

3 Big data security analytics techniques

Britain's digital spies have turned to Google for help making sense of the floods of data now inundating their powerful computing resources.

GCHQ, the Cheltenham-based signals intelligence agency, is recruiting an expert on MapReduce, the patented number-crunching technique previously behind the dominant web search engine.

The agency's new lead researcher on data mining will be responsible for "developing MapReduce analytics on parallel computing clusters", a job advertisment reveals.

MapReduce was developed by Google to index billions of web pages across its cluster of hundreds of thousands of commodity servers. It breaks up complicated tasks into smaller, easier computing problems that cheap hardware is capable of solving quickly.

Google patented the technique earlier this year, but it remains free for other organisations to adopt via Hadoop, an open source project. Originally described in a 2004 research paper, MapReduce has allowed Google's algorithms to index a rapidly expanding web while keeping costs down.

GCHQ faces similar a challenge as it gathers more and more raw data from internet communications, including email, social networks and VoIP.

"Successful data-driven organisations must be able to process, interpret and rapidly respond to indicators derived from unprecedented volumes of data from disparate information sources," its recruitment advertisement says.

The Register understands that GCHQ now has a cluster of more than 250,000 commodity servers under its Cheltenham "doughnut" building. In recent years it has developed this Google-style infrastructure instead of the very expensive, bespoke supercomputers it used to analyse microwave intercepts during the Cold War.

While spies are planning research on MapReduce, Google has already moved on to BigTable, its new distributed database. ®

Combat fraud and increase customer satisfaction

More from The Register

next story
Lavabit loses contempt of court appeal over protecting Snowden, customers
Judges rule complaints about government power are too little, too late
Don't let no-hire pact suit witnesses call Steve Jobs a bullyboy, plead Apple and Google
'Irrelevant' character evidence should be excluded – lawyers
Record labels sue Pandora over vintage song royalties
Companies want payout on recordings made before 1972
EFF: Feds plan to put 52 MILLION FACES into recognition database
System would identify faces as part of biometrics collection
Edward Snowden on his Putin TV appearance: 'Why all the criticism?'
Denies Q&A cameo was meant to slam US, big-up Russia
Ex-Tony Blair adviser is new top boss at UK spy-hive GCHQ
Robert Hannigan to replace Sir Iain Lobban in the autumn
Judge halts spread of zombie Nortel patents to Texas in Google trial
Epic Rockstar patent war to be waged in California
US Supreme Court supremo rakes Aereo lawman in oral arguments
Antenna-array content streamers: 'Ruling against us could dissipate the cloud'
German space centre endures cyber attack
Chinese code retrieved but NSA hack not ruled out
prev story

Whitepapers

Mobile application security study
Download this report to see the alarming realities regarding the sheer number of applications vulnerable to attack, as well as the most common and easily addressable vulnerability errors.
3 Big data security analytics techniques
Applying these Big Data security analytics techniques can help you make your business safer by detecting attacks early, before significant damage is done.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Securing web applications made simple and scalable
In this whitepaper learn how automated security testing can provide a simple and scalable way to protect your web applications.
Combat fraud and increase customer satisfaction
Based on their experience using HP ArcSight Enterprise Security Manager for IT security operations, Finansbank moved to HP ArcSight ESM for fraud management.