New Snowden leak: How NSA shared 850-billion-plus metadata records

'Federated search' spaffed info all over Five Eyes chums

Intelligent flash storage arrays

Documents leaked by Edward Snowden suggest during the noughties, the NSA massively expanded the information it shared with its Five Eyes allies and other agencies.

In slides given to The Intercept, the NSA boasts that its ICREACH program “increases NSA communications metadata sharing from 50 billion records to 850+ billion records”, with a footnote adding that 126 billion of these new records came from “2nd party SIGINT partners”.

At the same time, there was also a massive increase in the metadata the NSA was storing. The date, time, duration, called number and calling party – drawn from old PSTN systems – plus more than 20 new fields including latitude and longitude became part of the data-sharing.

To cope with all of this the NSA also needed to create a federated database search engine, in essence its own middleware sitting between the databases and the users.

Fast growth of NSA datasets

Massive increase in NSA's metadata volumes in 2006/2007. Image: The Intecept

The documents, dated 2006 and 2007, describe ICREACH as a “federated query” engine that would search “across all data sets for information relating to a target identifier.” The leaked documents say an agent could search from a single login to retrieve “phone number, global mobile satellite and cellular events and selectors, e-mail address, etc”.

Data brokers were also crafted to give the NSA's partners – the Australian Signals Directorate (then DSD), Britain's GCHQ, New Zealand's GCSB, Canada's CSE and the US FBI – access to the same search capabilities.

New fields added to what the NSA shared

The NSA shared more metadata fields post-ICREACH. Image: The Intercept

While maintaining the position that metadata is “information about content (but not the content itself)”, the NSA also notes that the data collected includes “formats and protocols used to render the information for people and systems”.

The middleware also meant that analysts were able to work on intelligence leads “without requiring access to raw intelligence”, the leaks claim. ®

Internet Security Threat Report 2014

More from The Register

next story
Facebook, Apple: LADIES! Why not FREEZE your EGGS? It's on the company!
No biological clockwatching when you work in Silicon Valley
Lords take revenge on REVENGE PORN publishers
Jilted Johns and Jennies with busy fingers face two years inside
Yes, yes, Steve Jobs. Look what I'VE done for you lately – Tim Cook
New iPhone biz baron points to Apple's (his) greatest successes
Happiness economics is bollocks. Oh, UK.gov just adopted it? Er ...
Opportunity doesn't knock; it costs us instead
Ex-US Navy fighter pilot MIT prof: Drones beat humans - I should know
'Missy' Cummings on UAVs, smartcars and dying from boredom
Sysadmin with EBOLA? Gartner's issued advice to debug your biz
Start hoarding cleaning supplies, analyst firm says, and assume your team will scatter
Facebook pays INFINITELY MORE UK corp tax than in 2012
Thanks for the £3k, Zuck. Doh! you're IN CREDIT. Guess not
Edward who? GCHQ boss dodges Snowden topic during last speech
UK spies would rather 'walk' than do 'mass surveillance'
prev story


Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Why and how to choose the right cloud vendor
The benefits of cloud-based storage in your processes. Eliminate onsite, disk-based backup and archiving in favor of cloud-based data protection.
Three 1TB solid state scorchers up for grabs
Big SSDs can be expensive but think big and think free because you could be the lucky winner of one of three 1TB Samsung SSD 840 EVO drives that we’re giving away worth over £300 apiece.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.