Feeds

Cleversafe cuddles up to MapReduce, kicks HDFS out of bed

Hey Hadoop, we think your filesystem sucks

HP ProLiant Gen8: Integrated lifecycle automation

Object storage specialist Cleversafe is after a piece of Big Data analytics action, and has wheeled in MapReduce to make it happen. In the same stroke it rejected HDFS as vulnerable and wasteful of storage capacity.

MapReduce is the analysis part of Hadoop, the flavour-of-the-month open-source data hoarding software. It processes information stored in Hadoop's Distributed Filesystem (HDFS), which employs a single metadata server and makes three copies of files to protect against drive failures and similar upsets.

Object storage systems such as Cleversafe's dsNet are designed to store masses of data on masses of connected nodes. It can self-heal by distributing metadata (and some kind of content hash) around the system of Slicestor nodes. Instead of adding HDFS capabilities alongside dsNet it's adding MapReduce to each Slicestor node.

Cleversafe says it stores only one copy of the MapReduce data instead of three, which is cheaper as capacities grow from terabytes to exabytes and on to petabytes. It isn't vulnerable to a catastrophic loss of data access caused by a metadata server failure because the metadata is distributed and protected - which is better than Hadoop according to Cleversafe.

There is no need for separate MapReduce servers and most Slicestor MapReduce apps will access data in their node rather than going out across the network so the system is more efficient.

Adding Cleversafe nodes adds capacity, both for raw data and metadata, and increases overall performance, the company claimed. The dsNet system can scale up into the petabyte range and beyond that into exabytes, it added.

Lockheed Martin is said to be working with Cleversafe to develop a suitable version of dsNet for US federal government agencies.

The initial MapReduce version of dsNet should be ready by the end of the year in a Cleversafe 3.0 software release. Support for third-party management tools - think Cloudera and HortonWorks - is expected to be added in the first half of 2013 in a 3.1 release. Conveniently, these builds may need Slicestore node hardware upgrades to cope with the added processing, memory and network loads. ®

Eight steps to building an HP BladeSystem

More from The Register

next story
THUD! WD plonks down SIX TERABYTE 'consumer NAS' fatboy
Now that's a LOT of porn or pirated movies. Or, you know, other consumer stuff
EU's top data cops to meet Google, Microsoft et al over 'right to be forgotten'
Plan to hammer out 'coherent' guidelines. Good luck chaps!
US judge: YES, cops or feds so can slurp an ENTIRE Gmail account
Crooks don't have folders labelled 'drug records', opines NY beak
Manic malware Mayhem spreads through Linux, FreeBSD web servers
And how Google could cripple infection rate in a second
FLAPE – the next BIG THING in storage
Find cold data with flash, transmit it from tape
Seagate chances ARM with NAS boxes for the SOHO crowd
There's an Atom-powered offering, too
prev story

Whitepapers

Seven Steps to Software Security
Seven practical steps you can begin to take today to secure your applications and prevent the damages a successful cyber-attack can cause.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Designing a Defense for Mobile Applications
Learn about the various considerations for defending mobile applications - from the application architecture itself to the myriad testing technologies.
Build a business case: developing custom apps
Learn how to maximize the value of custom applications by accelerating and simplifying their development.
Consolidation: the foundation for IT and business transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.