Feeds

NSA open sources Google database mimic

Crypto masters incubate Son of BigTable

Build a business case: developing custom apps

The US National Security Agency is open sourcing a distributed "NoSQL" database based on Google's proprietary BigTable platform.

Known as Accumulo, the platform has been in development at the NSA for over three years, and it's built atop Hadoop, the open source distributed file system and distributed number-crunching platform that mimics Google's internal infrastructure.

Unlike existing BigTable mimics such as HBase, Accumolo has "fine-grained" access controls and a new server-side programming mechanism that can modify data that's written to disk, or returned to the user. Using the cell-level access labels, you can provide external servers with access to some cells in the Accumolo data store but not others.

The NSA believes this may be of interest to government and health care operations and other outfits concerned with privacy. It acknowledges, however, that the access labels do not constitute a "complete security solution".

As noticed by H Online, the NSA has officially proposed Accumulo as an incubator project at Apache. Though the agency has little experience with public open source work, it says that the project has been "treated internally" as an open source project since its inception in 2008.

"We intend to strongly encourage the community to help with and contribute to the code. We will actively seek potential committers and help them become familiar with the codebase," the agency says. "We do not anticipate difficulty in operating under Apache's development process."

It does acknowledges, however, that the project overlaps with HBase.

"Accumulo and HBase are both based on the design of Google's BigTable, so there is a danger that potential users will have difficulty distinguishing the two or that they will not see an incentive in adopting Accumulo. There are a few key areas in which Accumulo differs from HBase. Some of the desired features of Accumulo could be incorporated into HBase, however the most important of these may be unlikely to be adopted," the agency continues, referring to both the cell-level access labels and the server-side programming mechanism.

"It is a possibility that the codebases will ultimately converge, but the number of differences at the current time warrants a separate project for Accumulo."

Google does not open source the software platforms underpinning its internal infrastructure. But in 2004, it published papers describing its GFS distributed file system and its MapReduce distributed number crunching platform, and these gave rise to the independent Hadoop, which resides at Apache. Google's BigTable paper followed in 2006, and this served as the basis for HBase as well as Acumulo.

According to the NSA, its project now spans over 200,000 lines of (mostly Java) code and hundreds of pages of documentation. It's built atop not only the core Hadoop platforms, but Apache Zookeeper (a means of managing distributed services) and Thift (a framework for developing services across multiple languages).

At this point, there is no indication that the database platform is part of a top secret NSA mission to plant a trojan horse on the machines of innocent open source mavens across the globe. But we'll keep you updated. ®

Maximizing your infrastructure through virtualization

More from The Register

next story
Whoah! How many Google Play apps want to read your texts?
Google's app permissions far too lax – security firm survey
Chrome browser has been DRAINING PC batteries for YEARS
Google is only now fixing ancient, energy-sapping bug
Do YOU work at Microsoft? Um. Are you SURE about that?
Nokia and marketing types first to get the bullet, says report
Microsoft takes on Chromebook with low-cost Windows laptops
Redmond's chief salesman: We're taking 'hard' decisions
EU dons gloves, pokes Google's deals with Android mobe makers
El Reg cops a squint at investigatory letters
Big Blue Apple: IBM to sell iPads, iPhones to enterprises
iOS/2 gear loaded with apps for big biz ... uh oh BlackBerry
OpenWRT gets native IPv6 slurping in major refresh
Also faster init and a new packages system
Google shows off new Chrome OS look
Athena springs full-grown from Chromium project's head
prev story

Whitepapers

Top three mobile application threats
Prevent sensitive data leakage over insecure channels or stolen mobile devices.
The Essential Guide to IT Transformation
ServiceNow discusses three IT transformations that can help CIO's automate IT services to transform IT and the enterprise.
Mobile application security vulnerability report
The alarming realities regarding the sheer number of applications vulnerable to attack, and the most common and easily addressable vulnerability errors.
How modern custom applications can spur business growth
Learn how to create, deploy and manage custom applications without consuming or expanding the need for scarce, expensive IT resources.
Consolidation: the foundation for IT and business transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.