Feeds

Facebook helps coders get their Rocks off with flash-wringing DB lib

Open-source persistent key-value store sits in your NAND, chugging data

Combat fraud and increase customer satisfaction

Facebook has developed an open-source database library that wrings more performance out of server-side flash storage than ever before, while building on previous technology published by Google.

The RocksDB embeddable persistent key-value store was announced by Facebook on Thursday, and gives developers a C++ library based on version 1.5 of Google's open-source LevelDB tech to supercharge databases manipulating data held on modern high-IO server flash. Facebook is storing almost a petabyte of data in it at the moment.

Potential apps that may find the tech useful include spam-detection programs that need fast access to memory, real-time graph-search queries, real-time Hadoop queries, message queues that can handle high numbers of inserts and deletes, or user-facing apps that store viewing history and state of a website, Facebook said.

LevelDB was developed by Google and released in 2011 to help it expose some of the tricks it had learned while building BigTable in a way that could be used by outside developers. Some people have adopted the technology, including Riak specialists Basho who serve up a tweaked version of the software to customers that have non-random access patterns.

"We have customers with very large datasets, performing at or very near the I/O subsystem's maximum is a requirement for most if not all of them. Continuing to provide that level of ingest while at the same time servicing the hot subset of the data in read requests is something that LevelDB is very good at doing," Basho technical product manager Greg Burd told us via email.

Now Facebook has hopped on the LevelDB tech as well, and substantially enhanced it.

"RocksDB software can fully utilize the IOPS offered by flash storage, making it perform faster than LevelDB across random read, write, and bulk uploads," Facebook wrote. "Modern storage devices support up to 100,000 random operations per second on a single flash storage card. If you stripe ten of these cards, you get about one million random operations per second. Today, RocksDB is able to run fast enough to avoid being the bottleneck when operating on this type of storage."

The tech works by storing keys and values as arbitrary byte streams. Writes go straight into an in-memory data structure called memtable, and are optionally also written to a logfile that is a sequentially written to main storage. Once memtable fills it up it is flushed into an sstfile on main storage and the logfile is deleted.

The tech has been seen achieving 10 times faster random writes and 30 per cent faster random reads than LevelDB, and seems to have better performance at scale than vanilla LevelDB when dealing with large memory footprints, Facebook said, in an architecture outline published along with the code on Github.

"RocksDB isn't a distributed database - rather the focus is on making an efficient, high-performance, single-node database engine," the company wrote.

The release of RocksDB follows the publication of the code for Facebook's Hive-beating Presto system in early November.

Though Facebook says it is publishing the code under a three-clause BSD license "because we think it will be useful beyond Facebook," there is likely another motivation at play here: if the company gets its tools into the wider IT ecosystem, it makes it easier for Facebook to hire new talent, and can outsource some components of ongoing development to the wider open-source community. ®

High performance access to file storage

More from The Register

next story
This time it's 'Personal': new Office 365 sub covers just two devices
Redmond also brings Office into Google's back yard
Batten down the hatches, Ubuntu 14.04 LTS due in TWO DAYS
Admins dab straining server brows in advance of Trusty Tahr's long-term support landing
Microsoft lobs pre-release Windows Phone 8.1 at devs who dare
App makers can load it before anyone else, but if they do they're stuck with it
Half of Twitter's 'active users' are SILENT STALKERS
Nearly 50% have NEVER tweeted a word
Oh no, Joe: WinPhone users already griping over 8.1 mega-update
Hang on. Which bit of Developer Preview don't you understand?
Internet-of-stuff startup dumps NoSQL for ... SQL?
NoSQL taste great at first but lacks proper nutrients, says startup cloud whiz
Windows 8.1, which you probably haven't upgraded to yet, ALREADY OBSOLETE
Pre-Update versions of new Windows version will no longer support patches
Microsoft TIER SMEAR changes app prices whether devs ask or not
Some go up, some go down, Redmond goes silent
Ditch the sync, paddle in the Streem: Upstart offers syncless sharing
Upload, delete and carry on sharing afterwards?
prev story

Whitepapers

Designing a defence for mobile apps
In this whitepaper learn the various considerations for defending mobile applications; from the mobile application architecture itself to the myriad testing technologies needed to properly assess mobile applications risk.
3 Big data security analytics techniques
Applying these Big Data security analytics techniques can help you make your business safer by detecting attacks early, before significant damage is done.
Five 3D headsets to be won!
We were so impressed by the Durovis Dive headset we’ve asked the company to give some away to Reg readers.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Securing web applications made simple and scalable
In this whitepaper learn how automated security testing can provide a simple and scalable way to protect your web applications.