Feeds

HDS CTO: Man, I could just throttle our array... er, in a good way

Fat NAS box does dedupe, when it's got a sec

Internet Security Threat Report 2014

Hitachi Data Systems has bunged primary deduplication into its network-attached storage (HNAS) kit and Hitachi Unified Storage (HUS) mid-range array. That's according to the company's chief technology officer Hu Yoshida.

HNAS is the hardware-accelerated filer HDS obtained when it bought BlueArc; the system relies on programmable chips (FPGAs) to speed up its operation. The HUS array file controllers use the BlueArc hardware engine and software. Yoshida said the deduplication feature can:

  • Be automated.
  • Deduplicate data in place rather than just incoming bytes.
  • Throttle back if the file-serving workload passes a threshold.
  • Use a cryptographic hash algorithm to ensure data integrity.
  • Dedupe the entire usable capacity of a filer - between 4PB and 8PB depending on the HNAS model - 256 terabytes at a time.

There is "an intelligent deduplication process that knows when new data is added and automatically starts up the deduplication engine(s) as long as the system is not busy", the CTO wrote on his company website. The system throttles back the dedupe if it's too busy responding to file read and write requests. The dedupe process looks at stored files and uses a database of hashes to identify chunks of data that are duplicated. They are then deleted and the reclaimed space made available for other data.

The dedupe has its hashing and chunking accelerated in the FPGA hardware, rather than pure software on the general-purpose CPU. Yoshida blogged this week: "A base hashing/chunking engine licence is included free of charge. Three additional hashing/chunking engines can be licensed; the increase in dedupe performance is nearly four-fold [with four engines]."

We interpret this as meaning that HUS files and objects can be deduped, but not HUS blocks.

Yoshida said one HDS customer deduped 1.2 million files in 16 minutes, but didn't reveal the net capacity benefit. Dedupe efficiency is said to be "comparable to other dedupe algorithms" and "the efficiency of dedupe depends on the dataset and file system block sizes". Quite so. The dedupe feature was introduced in January, we're told. ®

Beginner's guide to SSL certificates

More from The Register

next story
Docker's app containers are coming to Windows Server, says Microsoft
MS chases app deployment speeds already enjoyed by Linux devs
'Hmm, why CAN'T I run a water pipe through that rack of media servers?'
Leaving Las Vegas for Armenia kludging and Dubai dune bashing
'Urika': Cray unveils new 1,500-core big data crunching monster
6TB of DRAM, 38TB of SSD flash and 120TB of disk storage
Facebook slurps 'paste sites' for STOLEN passwords, sprinkles on hash and salt
Zuck's ad empire DOESN'T see details in plain text. Phew!
SDI wars: WTF is software defined infrastructure?
This time we play for ALL the marbles
Windows 10: Forget Cloudobile, put Security and Privacy First
But - dammit - It would be insane to say 'don't collect, because NSA'
Oracle hires former SAP exec for cloudy push
'We know Larry said cloud was gibberish, and insane, and idiotic, but...'
prev story

Whitepapers

Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Why cloud backup?
Combining the latest advancements in disk-based backup with secure, integrated, cloud technologies offer organizations fast and assured recovery of their critical enterprise data.
Win a year’s supply of chocolate
There is no techie angle to this competition so we're not going to pretend there is, but everyone loves chocolate so who cares.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?
Intelligent flash storage arrays
Tegile Intelligent Storage Arrays with IntelliFlash helps IT boost storage utilization and effciency while delivering unmatched storage savings and performance.