Feeds

Big data can be deduplicated

Isilon view disputed

Top three mobile application threats

Opinion Ronaldo Yamashita, an ESG analyst in Brazil, took exception to the Isilon view that big data is not compressible and sent El Reg his reasoning why he thinks this is the case.

When we talk about "text" big data, like logs or information gathered from different sources (e.g. web, credit agencies, Facebook), it is highly compressible. In fact, most data warehousing products use column-based compression to achieve high de-duplication ratios and to improve performance. After all, the fastest I/O is the I/O you don't have to make.

De-duplicated data results in better cache utilisation and less disk I/O. Deduplication is useful at any scale; just because most products can't handle it at high volumes doesn't mean it should not be used.

When we look at things from an overall storage perspective, and not just from the specialised database point of view, Rob Peglar's worries with metadata make sense. [But there are ways around this.]

Microsoft proposed a way to reduce RAM needs for deduplication in its research technology called ChunkStash. It allocates only 2 bytes per record in RAM.

The problem of replicating metadata between nodes, was solved by Scality, a startup which uses DHT (Distributed Hash Tables) to address metadata distribution. This is the same technique that's used by P2P (peer to peer) systems which handle vast petabyte-scale amounts of data.

Scality is not as efficient as Isilon from a performance perspective, but it shows a possible way to solve the problem.

NetApp is as 'performant' as Isilon and solved the problem in a simpler way, with no dedupe metadata replication. Deduplication happens in a single node and the cluster has some intelligence to place similar files together. It is good for performance and also for deduplication.

Databases such as Vertica and Greenplum can benefit from data locality too. They don't use global deduplication and achieve spectacular compression ratios.

Ocarina, the compression/deduplication company bought by Dell, even demonstrated how to get better compression from unexpected files, like images and videos. It can be used in the oil and gas industry for example, where it had long been thought impossible to achieve good compression ratios.

The way companies handle data may make it more compressible too. Jesse Jonas, from IBM, has a great presentation showing how stacking data ... is great for data reduction algorithms, as you don't need a person's name two times.

Compression and deduplication will play a key role in big data; it's all about economics. As ESG's Steve Duplessie pointed out, the next storage battle will be economic. If your system demands an order of magnitude more storage than competing suppliers' systems, it will struggle to compete. ®

High performance access to file storage

More from The Register

next story
This time it's 'Personal': new Office 365 sub covers just two devices
Redmond also brings Office into Google's back yard
Kingston DataTraveler MicroDuo: Turn your phone into a 72GB beast
USB-usiness in the front, micro-USB party in the back
Dropbox defends fantastically badly timed Condoleezza Rice appointment
'Nothing is going to change with Dr. Rice's appointment,' file sharer promises
Inside the Hekaton: SQL Server 2014's database engine deconstructed
Nadella's database sqares the circle of cheap memory vs speed
BOFH: Oh DO tell us what you think. *CLICK*
$%%&amp Oh dear, we've been cut *CLICK* Well hello *CLICK* You're breaking up...
Just what could be inside Dropbox's new 'Home For Life'?
Biz apps, messaging, photos, email, more storage – sorry, did you think there would be cake?
IT bods: How long does it take YOU to train up on new tech?
I'll leave my arrays to do the hard work, if you don't mind
Amazon reveals its Google-killing 'R3' server instances
A mega-memory instance that never forgets
prev story

Whitepapers

Top three mobile application threats
Learn about three of the top mobile application security threats facing businesses today and recommendations on how to mitigate the risk.
Combat fraud and increase customer satisfaction
Based on their experience using HP ArcSight Enterprise Security Manager for IT security operations, Finansbank moved to HP ArcSight ESM for fraud management.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Five 3D headsets to be won!
We were so impressed by the Durovis Dive headset we’ve asked the company to give some away to Reg readers.
SANS - Survey on application security programs
In this whitepaper learn about the state of application security programs and practices of 488 surveyed respondents, and discover how mature and effective these programs are.