ZFS gets inline dedupe
Switch it on and off at the dataset level
SaaS data loss: The problem you didn’t know you had
Sun's Zettabyte File System (ZFS) now has built-in deduplication, making it probably the most space-efficient file system there is.
There's a discussion of ZFS deduplication in a Sun blog, which says that chunks of data, such as a byte range or blocks or files, are checksummed with a hash function and any duplicate chunks will not be stored but instead reference this master chunk.
Sun says that backup data, virtual desktop images, and source-code repositories all have highly redundant data, and that deduplication can reduce disk usage to a fraction of the raw space needed.
File-level deduplication has the lowest processing overhead but is the least efficient method. Block-level dedupe requires more processing power, and is said to be good for virtual machine images. Byte-range dedupe uses the most processing power and is ideal for small pieces of data that may be replicated and are not block-aligned, such as e-mail attachments. Sun reckons such deduplication is best done at the application level since an app would know about the data.
ZFS provides block-level deduplication, using SHA256 hashing, and it maps naturally to ZFS's 256-bit block checksums. The deduplication is done inline, with ZFS assuming it's running with a multi-threaded operating system and on a server with lots of processing power. A multi-core server, in other words.
To turn it on you simply tell ZFS to dedupe a named storage pool, such as a silo, and datasets within it:
zfs set dedup=on silo
zfs set dedup=on silo/mydataset
zfs set dedup=off silo/yourdataset
With data sets containing redundant data, there's a disk-capacity benefit and a disk-write I/O benefit as redundant data isn't written to disk.
You can tell ZFS to do full byte comparisons rather than relying on the hash if you want full security against hash duplicates:
zfs set dedup=verify silo
You can go the other way and use a simpler hashing algorithm to reduce processing overhead and combine it with the verify function to increase overall dedupe speed:
zfs set dedup=fletcher4,verify silo
ZFS's deduplication scales to the size of the filesystem. Once the mapping tables are too large to fit in memory, then dedupe performance will decrease - here's a case where solid state storage might be a good idea.
The beauty of ZFS dedupe is that you don't need special storage arrays to deduplicate data. Ordinary arrays are quite acceptable, and its applicability at a data-set level means that you need only to deduplicate the datasets with redundant data and not the others.
As it is inline deduplication, throwing more processing cores and memory at it makes it go faster. We'll have to wait and see if GreenBytes switches to ZFS dedupe from the technology it's currently using. It will also be interesting to see how ZFS deduplication products compare performance-wise with specialised deduplication storage arrays. ®
COMMENTS
Is this really a win?
So you've taken a contiguous sequence of bytes (nice, for modern discs) and replaced it with several disjoint sequences (yuck, since modern discs are essentially serial devices). All in the name of saving a few bytes, when hard disc space has never been cheaper, and at the cost of some CPU and (I presume) some cache thrashing whilst the system figures it out, when the memory wall is the main throttle on performance on most systems these days.
Clever, but is it really useful, and is the cost in reliability (for assuredly this is a more complicated file system than one without the feature, and complexity breeds bugs) worth it?
@AC: ZFS - Sweet mother of Buddha
Anonymous Coward Posted 21:52 GMT post, "Isn't the whole freakin' *PURPOSE* of backups to duplicate data? So in case the "original" data gets deleted, destroyed, overwritten, etc, *YOU HAVE ANOTHER COPY?!?!!*"
Robustness and Data Integrity are at the core of ZFS !
- data CRC checking
- silent data corruption is corrected
- user selectable RAID1, RAID5, RAIDZ, RAIDZ2 redundancy
- user selectable [virtually] unlimited snapshots, to take as many historical backups as you want
All of these mechanisms take care of original data which may get: deleted, destroyed, overwritten, etc.
Now - all of this is possible with speeding virtualization for dozens, hundreds, or thousands of virtual machines off of a very small storage system, for any operating system, using a relatively small quantity of memory with dedup in ZFS... leave the user data on an external ZFS server, but with dedup, hundreds of disk images can reside on a very small piece of local or remote storage.
With compression, [virtually] unlimited snapshots, double parity, no RAID5 write hole, [virtually] unlimited volume size, native iSCSI support, native CIFS support, flash read acceleration, flash write acceleration, dedup nearly here, and the Lustre clustering in 2010... Absolutely nothing production quality is lining up to seriously compare with ZFS in the industry.
Anyone running virtualization under any other file system other than ZFS is really at a disadvantage...
Re: It's not the fall, it's the sudden stop..
OK, here goes at picking at your argument....
You trust dedupe for everything except backup, inferring your OK for live service but not for the backup copy? You trust the tape that you might go back to even though there is really limited crc protection there and not hing like sha256?
From the blog site quoted :
"When using a secure hash like SHA256, the probability of a hash collision is about 2^-256 = 10^-77 or, in more familiar notation, 0.00000000000000000000000000000000000000000000000000000000000000000000000000001.
For reference, this is 50 orders of magnitude less likely than an undetected, uncorrected ECC memory error on the most reliable hardware you can buy. "
IT infrastructure monitoring strategies
Agentless Backup is Not a Myth
Top 10 SIEM implementer’s checklist
Steps to Take Before Choosing a Business Continuity Partner
Enabling efficient data center monitoring
