How NVMe tamed the cowboy world of the flash card

The NVMe gang

That recognition lay behind the formation of the Non-Volatile Memory Express, or NVMe, initiative.

An industry consortium, known as the NVMe group, which has more than 80 members and is directed by a 13-company promoter group, published the v1.0 NVMe standard in March 2011. Intel had formed an NVMHCI working group in 2007.

Promotor group members included Cisco, Dell, EMC, Intel, LSI, Micron, Oracle, Samsung and SanDisk. The first products were introduced in 2012.

SATA drivers are based on the Advanced Host Controller Interface (AHCI), which was defined for the disk drive era, as it happens, by Intel. A comparison of ACHI and NVMe features shows that the PCIe driver standard can handle many more IO requests and is more efficient in its operation, requiring fewer CPU cycles.

For example, with AHCI each command requires four uncacheable register reads, each needing 2,000 CPU cycles. NVMe requires none. AHCI has a single queue for IO requests with up to 32 commands. NVMe has up to 64,000 queues, each with 64,000 commands, and is designed to scale over the next few years.

Inside a Linux storage stack a SCSI/SAS IO can take 6.0 μs and 19,500 CPU cycles, whereas an NVMe request has a latency of 2.8 μs and needs 9,100 cycles.

Flash is faster than disk, and PCIe is faster than SAS or SATA. These two things combine to make PCIe flash cards faster than SAS.SATA SSDs, with NVMe clearing the way for product introductions

IDT introduced NVMe chipsets in August 2012 and Samsung brought out an XS1715 enterprise PCIe flash card 11 months later. Others followed. For example HGST has an NVMe Ultrastar SN100 flash card that runs at 750,000 IOPS. Its latency is 20 µs.

There is wide operating system support, with Windows 8.1 and Windows Server 2012 R2 given native NVMe support by Microsoft. Windows 7 and Windows Server 2008 R2 also received the functionality.

V3.3 of the Linux kernel had an NVMe driver in March 2012, and has been updated since then. Chrome operating system supports NVMe as does Solaris.

Now it is generally inconceivable that any PCIe flash drive would not have NVMe support. Job done?

Double bypass

Not really, as, although the flash is on the PCIe card and accessed through a flash-optimised driver, it is still an external IO, requiring the host operating system's IO code stack, with its own latency, to be traversed. Two bypasses are being developed for that.

One is to put the flash even closer to the CPU by mounting the flash dies on memory DIMM sockets. Diablo Technologies has its MCS technology and this is used by SanDisk as its ULLtraDIMM flash, sold on via OEM deals with server manufacturers such as Lenovo, Huaweri and Supermicro.

System-level code is needed for this, but since only SanDisk is involved then it has the effective standard and for now driver proliferation is not a problem.

The second approach was pioneered by Fusion-io and has more system-level code produced to spoof the host operating system into treating the PCIe flash as a logical extension of memory.

Running applications needing to read and write data to/from this flash issue memory access Get and Put commands which are redirected to the flash.

There is no host IS IO stack code traversal and the IOs complete even faster, albeit at the expense of application code changes.

SanDisk acquired Fusion-io, and its Zeta Scale and NVMFS technology, based on Fusion-io’s NVMFS and originally called Direct File System, is available to provide this extra speed boost to PCIe flash access.

The beauty of NVMe is that it has been designed for a future of non-volatile memory devices that are faster than NAND and may use byte-level addressing instead of NAND’s pages or blocks.

NVMe drivers can support ReRAM or STT-RAM or other post-NAND technology products which ship with an NVMe interface. Shipping them without that interface would delay their adoption while drivers were produced. And, as more were produced, we would need driver standardisation, again.

Been there, done that, and NVMe is the result. It is a very successful result that is paving the way to widespread adoption of PCIe flash, both in servers and external to them, on a PCIe fabric enabling sharing of a central PCIe flash resource.

NVMe has brought discipline to a wild west of PCIe flash card drivers. ®