Original URL: http://www.theregister.co.uk/2012/04/20/step_changes/

Three questions Fusion-io's rivals face after flash API bombshell

Apps bypassing OS and disk to store hot data - chaos or breakthrough?

By Chris Mellor

Posted in Storage, 20th April 2012 07:29 GMT

Happy days are coming for the denizens of data centre technology areas. Applications are going to get such an incredible speed boost that they will become jet-propelled, turbo-charged, super-charged. The average app will virtually be able to stop doing disk I/O ever again... well, for most things.

It's a two-step improvement. The first step is the use of flash memory, dramatically improving storage I/O speed in terms of latency and IOPS. Latency is then cut even more as the flash memory moves from being implemented as disk-replacing SSDs in storage arrays, then as array controller flash, and then as PCIe-connected flash in servers, cutting out network links altogether.

But not quite. Even though the flash memory is now resident in the server, it is still treated as storage by the operating system. This means I/Os to it go through the operating system's disk I/O subsystem. But there is no need for that. Instead it should be treated as a subsidiary tier of memory, with data reads and writes done in roughly the same way.

Fusion-io has developed software, featuring Auto Commit and Atomic Writes, that performs this task for Fusion's ioMemory server flash. It is currently being made available as a developer's SDK. A software developer called Citrusleaf, which provides NoSQL database software, has already demonstrated 400,000 transactions a second from a single database server using it at the DEMO 2012 conference yesterday.

Fusion-io's CEO and chairman, David Flynn, said the effect of this was comparable to the transition from application software using tape as its online store to using disk drives.

Can we really imagine average application software never having to write directly to disk again? I believe we can. Disk will not go away, not at all. In fact we'll need more of the stuff because of the amount of data needing to be stored. But it will be the backing store, not the primary data, the working data set store; that will increasingly become flash memory used as a memory tier, not a storage tier.

Eventually this will migrate from servers to desktop and notebook systems, as operating system software will be amended to use a flash memory tier and sidestep its cumbersome disk I/O subsystem code.

So, let's step back a moment. Multi-core CPUs currently spend a lit of time waiting for apps in servers, virtualised or not, to complete disk I/O. They won't have to do that and servers will run more apps as their excess CPU cycles get soaked up. There will have to be system software written to ensure that the server flash memory tier is fed and watered with data from disk backing stores. This could be an extension of existing array tiering software or new software.

The focus of primary data storage will move to directly attached flash in servers; the PCIe-connected flash sold by Fusion-io with software to sidestep the host OS's disk I/O subsystem code; or to networked, shared flash arrays that use a PCIe fabric or InfiniBand link to have the same access speed as PCIe-connected flash actually in the server – and using the same cut-through software.

By host OS I mean either Windows, Linux or Unix or a hypervisor like VMware, KVM, Xenserver or Hyper-V.

Storage array vendors are at a disadvantage here. They need three things to play in this area:

These are the table stakes I think are necessary for storage array vendors to play in the server flash application speed-up game. Getting the ability to accelerate applications by factors of 5X to 20X is going to place storage vendors in a whole new pecking order. Application acceleration glory days are there for the taking.

1. Who is going to be the next flash storage vendor to get the cut-through software capability Fusion-io has pioneered? How about OCZ?
2. Which of the storage array vendors is going to be first to respond to the challenge Fusion-io is mounting on their primary data storage role? EMC, HP or Huawei perhaps?
3. Which one of the server vendors is going to be the first to produce a server system with hardware and software technology equivalent to that of Fusion-io? It could be IBM but might well be Oracle.

Comments to the Reg Forum, Step Changes topic please. ®