DDN: We've smashed bandwidth bottleneck. The IOPS will blind you
Meet our bonkers burst buffer-driven system
DDN’s FlashScale (14KXi) is a scale-up and scale-out all-flash and processor powerhouse that can deliver headline-grabbing IOPS and bandwidth numbers - fancy a billion IOPS?
This array has dual active:active controllers, each with a pair of 18-core Xeon E5 v4 Broadwell CPUs, up to 2TB of DDR4-2400 memory, and an embedded 48-lane, PCIe gen 3 x 8 internal fabric (8 x 8).
The head unit is a 4U box as seen with the GRIDScaler GS14K scale-out filer.
The speeds and feeds from this 4U box are:
- 6 million random 4K IOPS
- Up to 72 x 12Gbit/s dual port SAS or 48 x NVMe dual-port SSDs (as with the GS14K) and up to 24 x SAS SSDs or HDDs. These use 72 x 2.5-inch bays of which 48 are NVMe-capable
- <1msec latency with average 100 microsec reads from flash (for in-array apps)
- Connectivity options:
- 24 x 16 or 32 Gbit/s Fibre Channel
- Twelve Mellanox FDR (14.0625Gbit/s) or EDR (25.78125Gbit/s) InfiniBand
- 10, 40 or 100 GbitE
- Six Intel Omni-Path 100Gbit/s fabric or DDN OmniConnect
- RAID 6: 8+2, 4+2 RAID 5: 8+1, 4+1 RAID 1: 1+1
The controller box (SFA 14KXi) is a performance node and there are 4U capacity nodes. Capacity can be scaled up by adding up to 10 of these enclosures, each containing 84 SSDs, as with the GS14K. DDN says each capacity node stores 672TB raw, which is odd as no commercial SSD is available with (672/84=) 8TB.
DDN says 8TB SAS SSDs are coming in the third quarter of this year, as are 6.4TB NVMe SSDs.
DDN FlashScale graphic
It claims that it has overcome a 20GB/sec Intel bandwidth bottleneck, adding that there have been no effective bandwidth increases since Sandy Bridge and PXIe gen 3 maximum speed has not increased. Parallelising IO, as DataCore has found out, can reap huge bandwidth and performance advantages. When will the mainstream incumbents follow suit?
The operating system is SFAOS v3.0, a real-time OS with embedded file systems. We’re told processing is interrupt-free and that there is massive and parallel IO. The 14KXi has a cache and a burst buffer – think IME, which provides IO and parallel file system optimisation.
A few years ago there were storage architecture thoughts about bringing storage to compute or compute to storage. NVME over fabrics is a modern variant of the bring storage to compute idea. DDN calls FlashScale a hyper-converged appliance as it integrates compute, storage, networking and virtualisation in one box, but this is to hyper-converged Nutanix and Simplivity systems as a powerboat is to a sailing dinghy.
What DDN has done is to bring compute to storage by being capable of running some applications in the array, listing: OpenStack; IBM’s Spectrum Scale; SAS analytics; kx (financial services); Hadoop; Lustre; SQL Server and Oracle. We should think of some databases, parallel file systems and Big Data, not the generality of enterprise applications.
Applications that run inside FlashScale controller nodes will get faster access to data, with 100 microsecond read access quoted by DDN. We’re told this can lower external server and licensing costs.
The application areas mentioned are financial services, life science, manufacturing, oil and gas, government, web scale, analytics, database and HPC environments.
FlashScale can enlarge capacity by adding up to ten 4U SS8412 capacity nodes per dual-controller 4U head unit each with six million IOPS and 60GB/sec bandwidth. There is a support limit of 912 SSDs, and a 72 SSD controller and 10 x 84-SSD expansion units reaches that limit.
A rack of ten controllers has 60 million IOPS and 600GB/sec bandwidth. We understand controller nodes can store up to 576TB (again implying 8TB drives through dividing 576 by the 72 slots.)
A rack of 11 capacity nodes holds up to 7.3PB while a rack of 10 performance nodes holds 5.76PB.
DDN exclaims that a set of 17 FlashScale racks provides a headline-chasing one billion IOPS, 10GB/sec of bandwidth and 100PB of storage. We suggest this might be some of the more expensive IT equipment on the planet, needing 170 performance nodes to attain the one billion IOPS number. The company says it can cost $1/GB so 17 racks of 170 performance nodes would cost $97.92m.
This enormously powerful array bears comparison with EMC’s DSSD D5 and similar NVMe-over fabrics arrays, as well as WDC SanDisk’s InfiniFlash, and Pure’s FlashBlade.
FlashScale uses traditional Fibre Channel, Ethernet and InfiniBand network connectivity with no RDMA-type NVMe over fabrics connectivity to hosts. Unless applications are running directly inside FlashScale then they will be affected by network latency. On that basis DSSD, Apeiron, E8, Exelero and Mangstor shared flash storage arrays should outperform it in latency terms.
The comparison with SanDisk’s InfiniFlash IF700 box is ludicrous as FlashScale just blows it away; DDN saying IntelliFlash has 1 million IOPS compared to FlashScale’s six million, and 12GB/sec bandwidth compared to FlashScale’s 60GB/sec, and only supports eight connected servers via SAS connectivity.
The SanDisk system also uses proprietary flash modules, and DDN says commodity SSDs will develop much faster than proprietary flash modules from any one company.
FlashBlade is a developing rack-scale array product from Pure Storage. DDN says it has 15GB/sec bandwidth, a quarter of FlashScale’s, and is restricted to 40GbitE for host connectivity. It also uses proprietary flash modules instead of off-the-shelf SSDs.
DDN characterises FlashBlade as intended for low-medium performance with unstructured object data. FlashScale is for high-performance and latency-sensitive access to structured and unstructured data. There are no IOPS and latency numbers for FlashBlade though.
Availability and pricing
Alex Bouzari, DDN CEO and co-founder, said: “Flashscale delivers a 5x performance increase over other flash arrays for as low as $1/GB.”
That would imply a 672TB capacity node would cost from $672,000. The $0.05/IOPS figure suggests the six million IOPS controller would cost from $300,000.
FlashScale will be generally available in August 2016. Twenty expansion enclosures will be available in the future, almost doubling capacity. General DDN FlashScale information is available here. ®