All hard drive arrays will mutate into flashy faster hybrids
El Reg storage hacks stare at crystal balls
Blocks and Files All hard drive storage arrays will become flash hybrids, with non-volatile memory fitted internally or bolted on externally. These mutants will fight all-flash arrays with better data management and server flash cache integration.
The aim of the game is to get more revs out of the engines: applications should run faster, more virtual machines can be crammed in, and processors are kept busy executing code instead of waiting for the storage array to transfer data. All because flash can be faster at recalling information than spinning disks.
The array vendors have known about the hard drive IO bottleneck for years. The proliferation of multi-socket servers with multi-core CPUs means more apps and more requests for data from storage.
A good operating system kernel will schedule work for a processor core to chomp through while it waits for a disk controller to reply to a request, assuming the data isn't already in a RAM cache. However in busy IO-bound environments, CPUs can end up running out of threads and be forced to idle for longer than is desired.
The tactic of adding spindles to boost array response times is over. It's now a question of where do you put the flash: in servers, in all-flash arrays or in disk drive arrays, or all three places?
Traditional disk drive array vendors are being assaulted on three fronts: firstly, limited capacity all-flash array (AFA) vendors are offering massively faster IO rates as the cost-per-gigabyte of expensive flash approaches disk price tag levels, thanks to deduplication and compression. Secondly, there are new hybrid flash-with-disk arrays that offer near-AFA speed and hard drive array capacity.
The third attack comes from accelerators such as Avere, which offer flash-based caching devices that sit in front of the disk array, right smack in the data path, and accelerate requests.
The disk drive storage array vendors, at a tremendous performance disadvantage, have no choice but to do a NetApp and turn their arrays into hybrids too. That means adding a flash cache to the controller, adding flash as an extra tier and/or adding flash as a volume-level cache inside the array. NetApp is doing this with FlashCache and FlashPools respectively.
If array vendors don't hybridise their products by adding internal flash then customers could do it for them by stuffing a flash accelerator in front of their arrays.
Hybridised legacy arrays may not be as fast as designed-from-the-ground-up hybrid array designs nor all-flash arrays, but they do have one potential advantage: servers with existing tiering or virtual tiering (NetApp) technology can reuse these systems to fill flash caches with hot, regularly accessed, data and retire cold data back to the array.
The El Reg storage desk believes that all Dell, EMC, HDS, HP, IBM and NetApp drive arrays will eventually become NAND-disk hybrids, and have functionality added to manage the flow of data to and from server flash memory and their bulk capacity data stores.
They may well add their own all-flash array products, like EMC's Thunder, and have these manage the flow of cached data to and from traditional hard drive arrays. These vedors will also benefit if VMware, Microsoft and Xen add disk-to-flash caching management to their server hypervisors.
There may well be an opportunity for a startup to develop software to manage the loading and unloading of data to and from banks of hard disks and a networked all-flash array or hybrid array. ®