Feeds

NSA open sources Google database mimic

Crypto masters incubate Son of BigTable

High performance access to file storage

The US National Security Agency is open sourcing a distributed "NoSQL" database based on Google's proprietary BigTable platform.

Known as Accumulo, the platform has been in development at the NSA for over three years, and it's built atop Hadoop, the open source distributed file system and distributed number-crunching platform that mimics Google's internal infrastructure.

Unlike existing BigTable mimics such as HBase, Accumolo has "fine-grained" access controls and a new server-side programming mechanism that can modify data that's written to disk, or returned to the user. Using the cell-level access labels, you can provide external servers with access to some cells in the Accumolo data store but not others.

The NSA believes this may be of interest to government and health care operations and other outfits concerned with privacy. It acknowledges, however, that the access labels do not constitute a "complete security solution".

As noticed by H Online, the NSA has officially proposed Accumulo as an incubator project at Apache. Though the agency has little experience with public open source work, it says that the project has been "treated internally" as an open source project since its inception in 2008.

"We intend to strongly encourage the community to help with and contribute to the code. We will actively seek potential committers and help them become familiar with the codebase," the agency says. "We do not anticipate difficulty in operating under Apache's development process."

It does acknowledges, however, that the project overlaps with HBase.

"Accumulo and HBase are both based on the design of Google's BigTable, so there is a danger that potential users will have difficulty distinguishing the two or that they will not see an incentive in adopting Accumulo. There are a few key areas in which Accumulo differs from HBase. Some of the desired features of Accumulo could be incorporated into HBase, however the most important of these may be unlikely to be adopted," the agency continues, referring to both the cell-level access labels and the server-side programming mechanism.

"It is a possibility that the codebases will ultimately converge, but the number of differences at the current time warrants a separate project for Accumulo."

Google does not open source the software platforms underpinning its internal infrastructure. But in 2004, it published papers describing its GFS distributed file system and its MapReduce distributed number crunching platform, and these gave rise to the independent Hadoop, which resides at Apache. Google's BigTable paper followed in 2006, and this served as the basis for HBase as well as Acumulo.

According to the NSA, its project now spans over 200,000 lines of (mostly Java) code and hundreds of pages of documentation. It's built atop not only the core Hadoop platforms, but Apache Zookeeper (a means of managing distributed services) and Thift (a framework for developing services across multiple languages).

At this point, there is no indication that the database platform is part of a top secret NSA mission to plant a trojan horse on the machines of innocent open source mavens across the globe. But we'll keep you updated. ®

High performance access to file storage

More from The Register

next story
Android engineer: We DIDN'T copy Apple OR follow Samsung's orders
Veep testifies for Samsung during Apple patent trial
Windows 8.1, which you probably haven't upgraded to yet, ALREADY OBSOLETE
Pre-Update versions of new Windows version will no longer support patches
OpenSSL Heartbleed: Bloody nose for open-source bleeding hearts
Bloke behind the cockup says not enough people are helping crucial crypto project
Half of Twitter's 'active users' are SILENT STALKERS
Nearly 50% have NEVER tweeted a word
Windows XP still has 27 per cent market share on its deathbed
Windows 7 making some gains on XP Death Day
Internet-of-stuff startup dumps NoSQL for ... SQL?
NoSQL taste great at first but lacks proper nutrients, says startup cloud whiz
Microsoft lobs pre-release Windows Phone 8.1 at devs who dare
App makers can load it before anyone else, but if they do they're stuck with it
US taxman blows Win XP deadline, must now spend millions on custom support
Gov't IT likened to 'a Model T with a lot of things on top of it'
prev story

Whitepapers

Mainstay ROI - Does application security pay?
In this whitepaper learn how you and your enterprise might benefit from better software security.
Five 3D headsets to be won!
We were so impressed by the Durovis Dive headset we’ve asked the company to give some away to Reg readers.
3 Big data security analytics techniques
Applying these Big Data security analytics techniques can help you make your business safer by detecting attacks early, before significant damage is done.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Mobile application security study
Download this report to see the alarming realities regarding the sheer number of applications vulnerable to attack, as well as the most common and easily addressable vulnerability errors.