Related topics

Diablo: Get a load of this, PCIe flash-floggers. It's DIMM fine stuff

PCIe bottlenecks MLC flash when collecting garbage, upstart claims

Diablo chart 2

Diablo has a devilish message for PCIe flash users and suppliers; we’re coming to get you with better, faster, more consistently performing technology.

Diablo provides Memory Channel Storage (MCS) technology to SanDisk’s subsidiary SMART Storage. It’s OEMed on to Supermicro and also IBM (soon to be Lenovo) for its x86 servers as ULLtraDIMM technology.

Flash chips are supplied as DIMMs that plug into a server’s memory bus, which is closer to the CPU and DRAM in latency terms than PCIe-connected flash cards.

Diablo White Paper

Diablo's diatribe against PCIE flash bottlenecking

Diablo claims testing has shown that, regarding PCIe bandwidth, “end-to-end enterprise storage performance is truly governed by the presence or lack of bottlenecks within the system”.

Its white paper says there is “a critical shortcoming ... a pervasive bottleneck that limits both the performance and scalability of PCIe-based storage devices”.

It also says that many PCIe flash cards use a single big ASIC to do garbage collection – reclaiming cells with deleted data – across many flash chips at the same time. “The 'big ASIC' approach creates a bottleneck under heavy I/O load. This results in degraded latency as the controller ASIC is unable to keep pace with the increased computational burden.”

Diablo’s paper includes charts comparing SLC and MLC flash IOPS vs latency at various proportions of read and write IO, showing that MLC PCIe flash cards exhibit pronounced latency increases as the IOPS rate increases.

Diablo chart

The text says:

Compared to SLC flash, MLC has inherently lower endurance and also requires more error correction. Therefore wear-levelling, garbage collection, and error correction all become more prevalent and computationally intensive. The resulting media management creates a bottleneck in the MLC-based solution. Due to this bottleneck, references to high PCIe interface speeds can be misleading.

Because of this, it claims, in the case of the 70/30 read/write scenario in the chart above: “With Reads and Writes interleaved, the associated bottleneck affects all request submissions. PCIe interface capability becomes moot as, once again, only a small portion (less than 30%) of the PCIe bandwidth is actually being utilized.”

Unlike PCIe flash with a big ASIC bottleneck, MCS Flash DIMM accesses occur in parallel: “MCS DIMMs are populated on multiple memory controller channels and media management tasks are dispersed across those channels in a distributed fashion.”

Diablo compares its FlashDIMMS with PCIe MLC flash cards in the next chart:

Diablo chart 2

Diablo MCS flash vs PCIe MLC flash

The MCS system provides, it claims, “a dramatic improvement over a PCIe-based SSD” in Read performance … vastly superior write performance. … [and] dominates a mixed-workload comparison.”

It doesn’t name the PCIe flash card product and supplier, and Diablo clearly has a marketing axe to grind here. Its white paper will surely prompt rebuttals from PCIe flash card suppliers. Hello? Fusion-io? SanDisk? Yes, the same SanDisk with its PCIe Enterprise Solid State Accelerator cards. These Lightning PCIe SSAs “feature a proprietary controller to exponentially increase the ability to handle big data requests, alleviate I/O bottlenecks, and increase bandwidth."

The read-intensive version of these use MLC flash [PDF]. Diablo's hope is that a substantial amount of PCIe flash card spend will transfer to MCS flash spend making its revenues boom. ®

Sponsored: Driving business with continuous operational intelligence