Behold this golden era of storage startups. It's coming to an end
Potential for revolutionary advances is shrinking
Comment We are living in one of the most fascinating storage times with a great and rewarding war of storage access latency, but the major gains have already been won – and the scope for future advances is narrowing.
A previous storage revolution concerned space reclamation with deduplication, compression and thin provisioning mounting a war against excess space usage. That has run its course.
A relatively less significant space-reduction skirmish, not a war, is concerned with copy data reduction.
A war against external arrays consists of hyper-converged infrastructure and server SANs going head-to-head, but you still end up with an array, a virtual one.
Other ongoing skirmishes are to do with silo unification, simpler storage management, and moving data to the cloud. The public cloud is mounting its own subversive war against the very idea of on-premises storage.
Currently the most significant on-premises storage technology war, to my mind, is the one focussed on killing data access latency.
How can the assertion that future on-premises storage advances will be reduced in scope be justified? Three points: when shared storage arrays are accessed at PCIe flash speed, when non-volatile memory DIMMs are accessed at near DRAM speed, and when parallelised IO gets rid of CPU-bound IO waits, then what other scope is there for significant latency-killing storage data access advances?
These might be some of the things we see:
- SSDs abolish disk latency
- NVMe abolishes SATA and SAS stack latencies
- NVMe over fabrics will effectively abolish shared external storage array access latency
- Non-volatile DIMMs will vastly enlarge memory address space and reduce the need for storage IOs
Broadly speaking, we are moving external array access from millseconds to microseconds, and coming new technology non-volatile DIMMs will move access based on flash DIMMS in the microseconds to XPoint/ReRAM/NVRAM DIMM access in nanoseconds.
The upsurge in storage access latency killing technologies has been fantastic, and while a move from milliseconds to microseconds is fabulous, and a move from microseconds to nanoseconds is significant, moves from larger microsecond or nanosecond latencies to smaller ones are less meaningful.
Storage startups proposing technology to improve microsecond or nanosecond access latencies by a few tens of percentage points will not excite venture capitalists as much as orders of magnitude improvements.
Consequently, fewer storage startups will get funded and the golden age of storage creativity we have been enjoying for the past few decades is drawing to a close. I wish it weren't so. It has been, and still is, a gloriously impressive era of storage technology startups but the sand in the storage hourglass is running out. We're moving from revolutionary storage technology advances to incremental improvements.
Unless new sand gets put in then winter is coming, and the great game of storage technology thrones is drawing to a close. ®