Feeds

NSA open sources Google database mimic

Crypto masters incubate Son of BigTable

Top 5 reasons to deploy VMware with Tegile

The US National Security Agency is open sourcing a distributed "NoSQL" database based on Google's proprietary BigTable platform.

Known as Accumulo, the platform has been in development at the NSA for over three years, and it's built atop Hadoop, the open source distributed file system and distributed number-crunching platform that mimics Google's internal infrastructure.

Unlike existing BigTable mimics such as HBase, Accumolo has "fine-grained" access controls and a new server-side programming mechanism that can modify data that's written to disk, or returned to the user. Using the cell-level access labels, you can provide external servers with access to some cells in the Accumolo data store but not others.

The NSA believes this may be of interest to government and health care operations and other outfits concerned with privacy. It acknowledges, however, that the access labels do not constitute a "complete security solution".

As noticed by H Online, the NSA has officially proposed Accumulo as an incubator project at Apache. Though the agency has little experience with public open source work, it says that the project has been "treated internally" as an open source project since its inception in 2008.

"We intend to strongly encourage the community to help with and contribute to the code. We will actively seek potential committers and help them become familiar with the codebase," the agency says. "We do not anticipate difficulty in operating under Apache's development process."

It does acknowledges, however, that the project overlaps with HBase.

"Accumulo and HBase are both based on the design of Google's BigTable, so there is a danger that potential users will have difficulty distinguishing the two or that they will not see an incentive in adopting Accumulo. There are a few key areas in which Accumulo differs from HBase. Some of the desired features of Accumulo could be incorporated into HBase, however the most important of these may be unlikely to be adopted," the agency continues, referring to both the cell-level access labels and the server-side programming mechanism.

"It is a possibility that the codebases will ultimately converge, but the number of differences at the current time warrants a separate project for Accumulo."

Google does not open source the software platforms underpinning its internal infrastructure. But in 2004, it published papers describing its GFS distributed file system and its MapReduce distributed number crunching platform, and these gave rise to the independent Hadoop, which resides at Apache. Google's BigTable paper followed in 2006, and this served as the basis for HBase as well as Acumulo.

According to the NSA, its project now spans over 200,000 lines of (mostly Java) code and hundreds of pages of documentation. It's built atop not only the core Hadoop platforms, but Apache Zookeeper (a means of managing distributed services) and Thift (a framework for developing services across multiple languages).

At this point, there is no indication that the database platform is part of a top secret NSA mission to plant a trojan horse on the machines of innocent open source mavens across the globe. But we'll keep you updated. ®

Security for virtualized datacentres

More from The Register

next story
New 'Cosmos' browser surfs the net by TXT alone
No data plan? No WiFi? No worries ... except sluggish download speed
iOS 8 release: WebGL now runs everywhere. Hurrah for 3D graphics!
HTML 5's pretty neat ... when your browser supports it
Mathematica hits the Web
Wolfram embraces the cloud, promies private cloud cut of its number-cruncher
Mozilla shutters Labs, tells nobody it's been dead for five months
Staffer's blog reveals all as projects languish on GitHub
'People have forgotten just how late the first iPhone arrived ...'
Plus: 'Google's IDEALISM is an injudicious justification for inappropriate biz practices'
SUSE Linux owner Attachmate gobbled by Micro Focus for $2.3bn
Merger will lead to mainframe and COBOL powerhouse
iOS 8 Healthkit gets a bug SO Apple KILLS it. That's real healthcare!
Not fit for purpose on day of launch, says Cupertino
prev story

Whitepapers

Secure remote control for conventional and virtual desktops
Balancing user privacy and privileged access, in accordance with compliance frameworks and legislation. Evaluating any potential remote control choice.
Intelligent flash storage arrays
Tegile Intelligent Storage Arrays with IntelliFlash helps IT boost storage utilization and effciency while delivering unmatched storage savings and performance.
WIN a very cool portable ZX Spectrum
Win a one-off portable Spectrum built by legendary hardware hacker Ben Heck
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Beginner's guide to SSL certificates
De-mystify the technology involved and give you the information you need to make the best decision when considering your online security options.