ZFS gets inline dedupe

Switch it on and off at the dataset level

Next gen security for virtualised datacentres

Sun's Zettabyte File System (ZFS) now has built-in deduplication, making it probably the most space-efficient file system there is.

There's a discussion of ZFS deduplication in a Sun blog, which says that chunks of data, such as a byte range or blocks or files, are checksummed with a hash function and any duplicate chunks will not be stored but instead reference this master chunk.

Sun says that backup data, virtual desktop images, and source-code repositories all have highly redundant data, and that deduplication can reduce disk usage to a fraction of the raw space needed.

File-level deduplication has the lowest processing overhead but is the least efficient method. Block-level dedupe requires more processing power, and is said to be good for virtual machine images. Byte-range dedupe uses the most processing power and is ideal for small pieces of data that may be replicated and are not block-aligned, such as e-mail attachments. Sun reckons such deduplication is best done at the application level since an app would know about the data.

ZFS provides block-level deduplication, using SHA256 hashing, and it maps naturally to ZFS's 256-bit block checksums. The deduplication is done inline, with ZFS assuming it's running with a multi-threaded operating system and on a server with lots of processing power. A multi-core server, in other words.

To turn it on you simply tell ZFS to dedupe a named storage pool, such as a silo, and datasets within it:

zfs set dedup=on silo

zfs set dedup=on silo/mydataset

zfs set dedup=off silo/yourdataset

With data sets containing redundant data, there's a disk-capacity benefit and a disk-write I/O benefit as redundant data isn't written to disk.

You can tell ZFS to do full byte comparisons rather than relying on the hash if you want full security against hash duplicates:

zfs set dedup=verify silo

You can go the other way and use a simpler hashing algorithm to reduce processing overhead and combine it with the verify function to increase overall dedupe speed:

zfs set dedup=fletcher4,verify silo

ZFS's deduplication scales to the size of the filesystem. Once the mapping tables are too large to fit in memory, then dedupe performance will decrease - here's a case where solid state storage might be a good idea.

The beauty of ZFS dedupe is that you don't need special storage arrays to deduplicate data. Ordinary arrays are quite acceptable, and its applicability at a data-set level means that you need only to deduplicate the datasets with redundant data and not the others.

As it is inline deduplication, throwing more processing cores and memory at it makes it go faster. We'll have to wait and see if GreenBytes switches to ZFS dedupe from the technology it's currently using. It will also be interesting to see how ZFS deduplication products compare performance-wise with specialised deduplication storage arrays. ®

Gartner critical capabilities for enterprise endpoint backup

More from The Register

next story
The Return of BSOD: Does ANYONE trust Microsoft patches?
Sysadmins, you're either fighting fires or seen as incompetents now
Microsoft: Azure isn't ready for biz-critical apps … yet
Microsoft will move its own IT to the cloud to avoid $200m server bill
Shoot-em-up: Sony Online Entertainment hit by 'large scale DDoS attack'
Games disrupted as firm struggles to control network
Cutting cancer rates: Data, models and a happy ending?
How surgery might be making cancer prognoses worse
Silicon Valley jolted by magnitude 6.1 quake – its biggest in 25 years
Did the earth move for you at VMworld – oh, OK. It just did. A lot
Forrester says it's time to give up on physical storage arrays
The physical/virtual storage tipping point may just have arrived
prev story


Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
5 things you didn’t know about cloud backup
IT departments are embracing cloud backup, but there’s a lot you need to know before choosing a service provider. Learn all the critical things you need to know.
Why and how to choose the right cloud vendor
The benefits of cloud-based storage in your processes. Eliminate onsite, disk-based backup and archiving in favor of cloud-based data protection.
Top 8 considerations to enable and simplify mobility
In this whitepaper learn how to successfully add mobile capabilities simply and cost effectively.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?