Feeds

Could a hard drive dedupe data?

Manufacturers start looking for the next big thing

The essential guide to IT transformation

Comment Hitachi GST president Steve Milligan says one of the drivers affecting the hard drive industry is the need for efficient storage with technologies like virtualisation and deduplication. What is he on about?

He presented at a recent Needham conference for HDD investors and said that the storage market was driven by three things. First was the need to store more data, which he saw continuing if not accelerating. Then he said mobility had risen strongly and was set to continue as people needed to store more data in mobile devices. Thirdly, he mentioned storage efficiency and virtualisation and deduplication.

Why would a hard drive manufacturer be bothered about these two items? All HGST and the other drive suppliers do is build a raw byte storing device and present it to drive array and compute device manufacturers as a component. They add virtualisation functions and may build the drives into deduplicating storage arrays. Deduplication simply isn't a concern of drive manufacturers, or is it?

A couple of years ago, data encryption was performed by in-band devices, such as NetApp's Decru boxes, applications running in a server, or tape drives that encrypted backup data in case the tape cartridges got lost or stolen.

As encryption hardware was then small enough to shove in a tape drive controller, it was also small enough to put inside a hard drive's case. So Seagate and others came out with self-contained hard drive encryption products.

Could Milligan be thinking that where an encryption ASIC can go then a deduplication ASIC could follow?

You would need to do HDD-level dedupe inside the HDD case, where space is restricted, and you'd need to provide CPU cycles because the HDD controller CPU is already busy doing standard HDD I/O-type tasks. Suppose there were an ASIC, an application-specific processor you could use?

Exar has bought Hifn for its hardware encryption, compression and deduplication products. Here is a description of a Hifn chip:

The 9725 provides power efficient hardware offload capabilities for data compression, encryption and deduplication for enterprise class storage applications. OEMs now have the ability to integrate these capabilities into platforms that require non-standard PCIe form factors or need to be embedded directly on a system’s motherboard.

Solid State Drive start-up Whiptail is adding Hifn dedupe to its RaceRunner SSDs, so the idea isn't entirely half-arsed.

As storage array futures look to be heading to two main disk tiers, one for performance (FC, SAS, SSD) and one for capacity (SATA) then taking a 2TB SATA drive used for virtual tape library and disk-to-disk backup applications and increasing its capacity by 20, 30 or possibly more per cent by deduplicating data stored on it, with no decrease in I/O performance, might look interesting to a hard drive manufacturer looking to gain an edge and justify a pricing differential.

By the way, for a vendor of sealed disk drive canisters, aka Integrated Storage Elements (ISEs), like Xiotech, adding this functionality at a canister level might be worth considering.

Of course Milligan could just have been talking generally. But it's enticing isn't it? Once HDDs didn't encrypt their data. Now they do because there is enough HDD controller board real estate to permit the addition of encryption hardware. What else might be added, what other functions might be considered by the HDD suppliers? Could on-board - or in-canister - HDD deduplication be a realistic prospect over the next few years? ®

Boost IT visibility and business value

More from The Register

next story
Pay to play: The hidden cost of software defined everything
Enter credit card details if you want that system you bought to actually be useful
Shoot-em-up: Sony Online Entertainment hit by 'large scale DDoS attack'
Games disrupted as firm struggles to control network
HP busts out new ProLiant Gen9 servers
Think those are cool? Wait till you get a load of our racks
Silicon Valley jolted by magnitude 6.1 quake – its biggest in 25 years
Did the earth move for you at VMworld – oh, OK. It just did. A lot
VMware's high-wire balancing act: EVO might drag us ALL down
Get it right, EMC, or there'll be STORAGE CIVIL WAR. Mark my words
Forrester says it's time to give up on physical storage arrays
The physical/virtual storage tipping point may just have arrived
prev story

Whitepapers

Top 10 endpoint backup mistakes
Avoid the ten endpoint backup mistakes to ensure that your critical corporate data is protected and end user productivity is improved.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Backing up distributed data
Eliminating the redundant use of bandwidth and storage capacity and application consolidation in the modern data center.
The essential guide to IT transformation
ServiceNow discusses three IT transformations that can help CIOs automate IT services to transform IT and the enterprise
Next gen security for virtualised datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.