We pick a storage CTO's brains on Linux-heads, big vendors – and should all the admins NVMe?

Interview NVMe storage networking (NVMe-oF or NVME over fabrics) promises to radically cut the latency of access to external block storage compared to iSCSI and Fibre Channel.

It works by implementing an RDMA (Remote Direct Memory Access) request to an external storage drive, bypassing the traditional host server and source array storage IO stacks.

It has to be implemented across network or fabric and, so far, Ethernet, with ROCE (RDMA over Converged Ethernet), is the favoured one – with InfiniBand in second place. However, it can be implemented across a Fibre Channel link as well (FC-NVMe), and both Brocade and Cisco, which dominate Fibre Channel storage networking, are developing FC-NVMe support in their directors, switches and host bus adapters.

On the face of it, adding FC-NVMe to existing Fibre Channel products looks to be an obvious thing to do but what does an old Fibre Channel and storage networking hand think about NVMe over Fibre Channel and ordinary NVMe over Ethernet?

Greg Scherer has spent almost 40 years in tech, mostly involved with data centre IO and OS integration. He was the CTO and SVP for business development at Emulex, joining in 2000 and leaving in 2006. He joined Neterion as its CTO in 2007 and joined Broadcom as its VP for server and storage strategy in 2009.

He became VP for engineering product strategy at Cavium-owned QLogic in 2014, moving on to be the CTO in late 2014, and then CTO for Ethernet and Fibre Channel Adapters in 2016, leaving for semi-retirement at the end of 2017.

FC-NVMe is ideal for existing enterprises that already employ Fibre Channel, where NVMe-oF over RDMA Ethernet is best suited for green-field "Linux" environments...

He tells us: "I was ... involved with Fibre Channel since 'quarter-speed' (250Mbit/s) and Ethernet since 10Mbit/s, so I'm very familiar with the speeds, encoding and history of both."

El Reg: What are your basic thoughts on Ethernet-based and Fibre Channel-based NVMe networking?

Greg Scherer: I'm pretty bullish about NVMe-oF and FC-NVMe; when I was with QLogic/Cavium my group was responsible for both Ethernet and Fibre Channel futures.  Also, the chair of the FC-NVMe working group (Craig Carlson) was in my group, so I've been pretty deep into FC-NVMe.

El Reg: Are the deployment cases the same?

Greg Scherer: I think the deployment cases for NVMe-oF over RDMA Ethernet and FC-NVMe are very different from one another; FC-NVMe is ideal for existing enterprises that already employ Fibre Channel, where NVMe-oF over RDMA Ethernet is best suited for green-field "Linux" environments.

El Reg: Does FC-NVMe have any particular relevant attributes here?

Greg Scherer: FC-NVMe has several key advantages that make it Enterprise-ready to deploy. It uses a single fabric-resident discovery mechanism by using the Fibre Channel NameServer.

This means that it's possible to deploy FC-NVMe in a true heterogeneous (diverse OS) environment; every OS (Windows, Linux, VMWare, etc.) will use the "same" mechanism for device discovery. The Fabric allows for "zoning" through the discovery process, so out of the box, a SAN admin can fence different servers to only see specific devices.

Since the low-level driver isn't a part of the OS, like NVMe-oF, but rather supplied by the HBA vendor (Emulex or QLogic), the deployment for OSes outside the Linux model should be more agile... the market doesn't have to wait until VMware, Microsoft and others deploy their NVMe-oF ecosystem.

El Reg: Do you have a view on single standards?

Greg Scherer: NVMe-oF using RDMA Ethernet is also a wonderful technology, but I believe made the same mistake as iSCSI by not creating a SINGLE standard for device discovery; the spec allows device discovery to be "optional", but does include multiple possible methods; this means that any two OS implementations aren't guaranteed to use the same method and therefore interoperability is questionable at best.

El Reg: Does NVMe-oF have any disadvantages compared to FC-NVMe?

Greg Scherer: NVMe-oF – or RDMA Ethernet – is a great technology but is hindered in wide-scale deployment today by several limitations:

1. You need a homogeneous (single OS) Linux environment
2. You must run a recent version of Linux; no other OSes have NVMe-oF drivers available (yet)
3. You have a "closed" environment and can control how/who talks to NVMe-oF devices
4. It's a closed environment – [a] single vendor, single use-case environment, typically running Linux

El Reg: What deployment area would fit these limitations?

Greg Scherer: I know of several storage array companies that are exploring/planning to replace SAS with NVMe-oF on the back-end. This fits the current limitations very well.

[Also] the "public cloud" likes NVMe-oF because they can build cheap Ethernet servers with shelves of NVMe drives that are shared by multiple Linux servers over NVMe-oF. These are also closed ecosystems, with single policies about who/how storage is used.

+Comment

Given that FC-NVMe looks to be a natural upgrade path to NVMe fabric connectivity for SANs it would be good if there was a "travel guide" for the journey, an FC-NVMe "cookbook".

We're asking Greg what he thinks might be involved when an FC SAN user decides to move to FC-NVMe connectivity for external storage. Any scribes of FC-NVMe geek's guides are welcome to get in touch. ®