Deduplication: a power-hungry way to streamline storage
Data wants to stay single
Windows Server 8 is coming, and it is bringing storage enhancements with it.
Data deduplication in particular has caught my eye: it is something I have wanted on my Windows file servers for a long time.
This technology is nothing new; ZFS has had deduplication for a while now, and this technology is (experimentally) available with Linux’s Btrfs as well.
Worth consideration too is Opendedup, bringing deduplication to both Windows and Linux via SDFS.
The quick and dirty on deduplication is that it is an umbrella term for a set of technologies that allow you to store only one copy of a given piece of data on your hard drive, thus saving space and potentially speeding file writes. Essentially, it is single instance storage.
Deduplication can be done at the file level, the block level or the byte level. File and block level are the most common.
Need for speed
It can be done synchronously (as the writes happen) or asynchronously (as a scheduled job during quiet hours.)
Synchronous deduplication takes a lot of CPU power. So much power that high-end filer manufacturers are always clamouring for the fastest possible Xeons, and are pushing forward with research into making use of GPGPU technology.
It’s easy to imagine why. Try to compress 5GB of text files into a zip ball. Now, picture your hard drive as a half-petabyte zip ball that you are reading from and writing to at 10Gbit/s. Processing power is suddenly very important.
Despite this, deduplication is a critical technology. Storage demand has consistently outpaced capacity growth. What’s more, while hard drive capacity has trebled, network I/O and disk speeds have not.
This has potentially disastrous implications for both Raid rebuild times and backups. Deduplication can reduce the amount of information to Raid or backup, helping to ensure both of these processes occur in timeframes compatible with business needs.
Risky business
This is assuming that you are backing up the deduplicated blocks instead of the full file set. There are arguments for and against both.
Backing up the deduplicated blocks means less backup media is required and less bandwidth has to be set aside to perform the backups. On the other hand, it can increase restore times dramatically, as the entire set of backup media is now hopelessly interdependent.
Most people won’t back up data as deduplicated blocks – it is just too risky. The loss of one piece of backup media can render data irretrievable on all other media. This means budgeting backup bandwidth for the fully undeduplicated data to run every night.
You also have to budget your storage I/O bandwidth for the undeduplicated data size, not the size as it is stored on disk. The amount of data on disk may change only by a few dozen gigabytes a day, but the total storage I/O off that system could be measured in dozens of terabytes.
Mind the gap
Deduplication is necessary, increasingly so as the gap between storage demand and availability grows. But it doesn’t help decrease the need for network bandwidth, and it imposes a hefty processing requirement.
My next filer looks like its going to have a pair of top end Xeons and 10GBE. It will need to have two 10GbE ports, as I need to allow for MPIO.
Factor in sizing the filer to deal with demand peaks, ability to support snapshots, previous versions and other fun features, and the thought of planning my next storage refresh gives me a headache.
Difficult or no, time must be taken to do the research. The cost of storage and its attendant networking is such that few among us can afford to get it wrong. ®
Regcast training : Hyper-V 3.0, VM high availability and disaster recovery
COMMENTS
Not as common/ready as you may think...
The deduplication functionality in BTRFS is completely useless, it is at a pre-experimental stage, and isn't (and judging by the discussions I had on the mailing list likely never will be) synchronous like in all other sensible dedupe implementations. So don't bank on that. But - ZFS has been available on Linux for a while, with both kernel level and fuse implementations available.
Another option for deduplication is lessfs (fuse based).
Another thing worth mentioning is that synchronous deduplication is always going to be cheaper in terms of total CPU and disk I/O terms than asynchronous deduplication because you end up having to do the disk I/O twice. Conceptually, this is similar to the parity RAID (5, 6) write-hole (write-read-checksum).
>Do a lookup in an in-memory database of that checksum. If you already have that data, you can
>stop - there is no need to actually write the block to the disk, saving you power. If you don't already
>have that data, save it as usual.
You missed out the important step (which is what makes async and sync dedupe more similar in resource than you think).
If your checksum shows a *possible* duplicate block, you bit-by-bit compare the two data blocks to make sure they really are identical. Which involves reading the potentially identical block from disk ofc.
Then, if they are *actually* identical, you write a pointer instead of a duplicate of the data.
Presumably an async dedupe process would need some way of marking up non-identical blocks with checksum collisions to prevent constant comparisons.
If you have a virtualized environment, with 5 or more instances of an OS on a single system - dedup makes a lot of sense.
Dedup built into the underlying OS (ZFS) providing the virtualization also means the expensive memory serving as the disk cache (ARC) is also deduped... not to mention any expensive blocks of storage sitting in flash memory (L2ARC) is also deduped.
To understand the benefits of dedup (reduction in cost of memory, flash, storage, I/O) for increased throughput, at the expense of a needing some more CPU cores - the tradeoff is a no-brainer, especially since ZFS is free and already bundled with multiple OS's.
An extra CPU core is cheap in comparison to memory, flash, and disk storage while no other special hardware controllers are needed (at least for ZFS.)
IT infrastructure monitoring strategies
Requirements Checklist for Choosing a Cloud Backup and Recovery Service Provider
Cloud based data management
Enabling efficient data center monitoring
Agentless Backup is Not a Myth
