Feeds

HDS CTO: Man, I could just throttle our array... er, in a good way

Fat NAS box does dedupe, when it's got a sec

The Essential Guide to IT Transformation

Hitachi Data Systems has bunged primary deduplication into its network-attached storage (HNAS) kit and Hitachi Unified Storage (HUS) mid-range array. That's according to the company's chief technology officer Hu Yoshida.

HNAS is the hardware-accelerated filer HDS obtained when it bought BlueArc; the system relies on programmable chips (FPGAs) to speed up its operation. The HUS array file controllers use the BlueArc hardware engine and software. Yoshida said the deduplication feature can:

  • Be automated.
  • Deduplicate data in place rather than just incoming bytes.
  • Throttle back if the file-serving workload passes a threshold.
  • Use a cryptographic hash algorithm to ensure data integrity.
  • Dedupe the entire usable capacity of a filer - between 4PB and 8PB depending on the HNAS model - 256 terabytes at a time.

There is "an intelligent deduplication process that knows when new data is added and automatically starts up the deduplication engine(s) as long as the system is not busy", the CTO wrote on his company website. The system throttles back the dedupe if it's too busy responding to file read and write requests. The dedupe process looks at stored files and uses a database of hashes to identify chunks of data that are duplicated. They are then deleted and the reclaimed space made available for other data.

The dedupe has its hashing and chunking accelerated in the FPGA hardware, rather than pure software on the general-purpose CPU. Yoshida blogged this week: "A base hashing/chunking engine licence is included free of charge. Three additional hashing/chunking engines can be licensed; the increase in dedupe performance is nearly four-fold [with four engines]."

We interpret this as meaning that HUS files and objects can be deduped, but not HUS blocks.

Yoshida said one HDS customer deduped 1.2 million files in 16 minutes, but didn't reveal the net capacity benefit. Dedupe efficiency is said to be "comparable to other dedupe algorithms" and "the efficiency of dedupe depends on the dataset and file system block sizes". Quite so. The dedupe feature was introduced in January, we're told. ®

The Essential Guide to IT Transformation

More from The Register

next story
Sysadmin Day 2014: Quick, there's still time to get the beers in
He walked over the broken glass, killed the thugs... and er... reconnected the cables*
Auntie remains MYSTIFIED by that weekend BBC iPlayer and website outage
Still doing 'forensics' on the caching layer – Beeb digi wonk
Microsoft says 'weird things' can happen during Windows Server 2003 migrations
Fix coming for bug that makes Kerberos croak when you run two domain controllers
Multipath TCP speeds up the internet so much that security breaks
Black Hat research says proposed protocol will bork network probes, flummox firewalls
Cisco says network virtualisation won't pay off everywhere
Another sign of strain in the Borg/VMware relationship?
Forrester says Australia, not China, is next boom market for cloud
It's cloudy but fine down under, analyst says
prev story

Whitepapers

Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Why and how to choose the right cloud vendor
The benefits of cloud-based storage in your processes. Eliminate onsite, disk-based backup and archiving in favor of cloud-based data protection.
The Essential Guide to IT Transformation
ServiceNow discusses three IT transformations that can help CIO's automate IT services to transform IT and the enterprise.
Maximize storage efficiency across the enterprise
The HP StoreOnce backup solution offers highly flexible, centrally managed, and highly efficient data protection for any enterprise.