Feeds

Big data can be deduplicated

Isilon view disputed

HP ProLiant Gen8: Integrated lifecycle automation

Opinion Ronaldo Yamashita, an ESG analyst in Brazil, took exception to the Isilon view that big data is not compressible and sent El Reg his reasoning why he thinks this is the case.

When we talk about "text" big data, like logs or information gathered from different sources (e.g. web, credit agencies, Facebook), it is highly compressible. In fact, most data warehousing products use column-based compression to achieve high de-duplication ratios and to improve performance. After all, the fastest I/O is the I/O you don't have to make.

De-duplicated data results in better cache utilisation and less disk I/O. Deduplication is useful at any scale; just because most products can't handle it at high volumes doesn't mean it should not be used.

When we look at things from an overall storage perspective, and not just from the specialised database point of view, Rob Peglar's worries with metadata make sense. [But there are ways around this.]

Microsoft proposed a way to reduce RAM needs for deduplication in its research technology called ChunkStash. It allocates only 2 bytes per record in RAM.

The problem of replicating metadata between nodes, was solved by Scality, a startup which uses DHT (Distributed Hash Tables) to address metadata distribution. This is the same technique that's used by P2P (peer to peer) systems which handle vast petabyte-scale amounts of data.

Scality is not as efficient as Isilon from a performance perspective, but it shows a possible way to solve the problem.

NetApp is as 'performant' as Isilon and solved the problem in a simpler way, with no dedupe metadata replication. Deduplication happens in a single node and the cluster has some intelligence to place similar files together. It is good for performance and also for deduplication.

Databases such as Vertica and Greenplum can benefit from data locality too. They don't use global deduplication and achieve spectacular compression ratios.

Ocarina, the compression/deduplication company bought by Dell, even demonstrated how to get better compression from unexpected files, like images and videos. It can be used in the oil and gas industry for example, where it had long been thought impossible to achieve good compression ratios.

The way companies handle data may make it more compressible too. Jesse Jonas, from IBM, has a great presentation showing how stacking data ... is great for data reduction algorithms, as you don't need a person's name two times.

Compression and deduplication will play a key role in big data; it's all about economics. As ESG's Steve Duplessie pointed out, the next storage battle will be economic. If your system demands an order of magnitude more storage than competing suppliers' systems, it will struggle to compete. ®

Reducing security risks from open source software

More from The Register

next story
Sysadmin Day 2014: Quick, there's still time to get the beers in
He walked over the broken glass, killed the thugs... and er... reconnected the cables*
SHOCK and AWS: The fall of Amazon's deflationary cloud
Just as Jeff Bezos did to books and CDs, Amazon's rivals are now doing to it
Amazon Reveals One Weird Trick: A Loss On Almost $20bn In Sales
Investors really hate it: Share price plunge as growth SLOWS in key AWS division
US judge: YES, cops or feds so can slurp an ENTIRE Gmail account
Crooks don't have folders labelled 'drug records', opines NY beak
Auntie remains MYSTIFIED by that weekend BBC iPlayer and website outage
Still doing 'forensics' on the caching layer – Beeb digi wonk
BlackBerry: Toss the server, mate... BES is in the CLOUD now
BlackBerry Enterprise Services takes aim at SMEs - but there's a catch
The triumph of VVOL: Everyone's jumping into bed with VMware
'Bandwagon'? Yes, we're on it and so what, say big dogs
Carbon tax repeal won't see data centre operators cut prices
Rackspace says electricity isn't a major cost, Equinix promises 'no levy'
prev story

Whitepapers

Designing a Defense for Mobile Applications
Learn about the various considerations for defending mobile applications - from the application architecture itself to the myriad testing technologies.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Top 8 considerations to enable and simplify mobility
In this whitepaper learn how to successfully add mobile capabilities simply and cost effectively.
Seven Steps to Software Security
Seven practical steps you can begin to take today to secure your applications and prevent the damages a successful cyber-attack can cause.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.