Original URL: http://www.theregister.co.uk/2013/04/23/mellanox_connectx_3_server_adapters/

Mellanox adds VM-flitting to ConnectX-3 server adapters

Going Pro with VXLAN and NVGRE to stretch across Layer 3 nets

By Timothy Prickett Morgan

Posted in Cloud, 23rd April 2013 16:49 GMT

Switch and adapter maker Mellanox Technologies added support for virtual LAN overlays to its Ethernet switches late last year and now it is building in support for the two primary overlays into its ConnectX-3 server adapters to match.

Virtual LAN overlays make Layer 3 routing between networks invisible to those networks and look like one, big, flat virtual network as far as virtual machines are concerned. This overlay allows for a virtual machine running on one server on one LAN to be live migrated across Layer 3 links to another server running on another LAN without screwing up or having to reset the network settings for that virtual machine.

Mellanox added support for virtual LAN overlays into its MLNX-OS switch operating system to allow routing over Layer 3 in the network back in November 2012. This Layer 3 overlay was enabled in any switch using the SwitchX or SwitchX-2 ASICs from Mellanox that was configured to support 10Gb/sec or 40Gb/sec Ethernet. Mellanox wove the VXLAN approach to virtual LAN overlays from VMware into its switch operating system, and also added the competing NVGRE approach from Microsoft to MLNX-OS.

You need virtual LAN overlays on a cloud for a simple reason. Virtual LANs top out at 4,096 in a single domain, and if you are building a cloud, you need a virtual LAN for every customer and their applications. In very large multitenant clouds, you quickly run out of virtual LANs.

The ConnectX-3 Pro server adapter from Mellanox

The ConnectX-3 Pro server adapter from Mellanox

More importantly, perhaps, when a virtual machine tries to jump from one network to another, it actually cuts itself off from the network, which is a perfect way to make a VM useless but also perfectly safe. The virtual LAN overlays create a tunnel that decouples a VM's location in a cluster from its address on the network.

This is all good, but at the moment, network adapter cards for servers also do a lot of different functions, such as TCP segmentation or TCP checksum offload, to speed up the network, but this functionality is all unavailable to the network once you use VXLAN or NVGRE.

Well, until now that is. The new ConnectX-3 Pro cards use a new chip etched by Mellanox that includes an interconnect offload engine so you can do virtual LAN overlays in hardware on the server adapter instead of imposing overhead on the server itself, and do so without sacrificing those other offload functions that are in the adapter.

Both VXLAN and NVGRE offload is supported on the new Pro adapter cards, matching what is available on the Ethernet switches from Mellanox that were tweaked last year to support virtual LAN overlays.

How the virtual LAN overlay plays out on servers and switches

How the virtual LAN overlay plays out on servers and switches

The ConnectX-3 Pro cards plug into PCI-Express 3.0 slots and come in single-port and dual-port variants supporting InfiniBand running at up to 56Gb/sec and Ethernet running at 40Gb/sec.

Specifically, there will be variants of the Pro cards that have one or two 10Gb/sec Ethernet ports, one or two 40Gb/sec ports, or one or two hybrid ports that can speak either 56Gb/sec InfiniBand or 40Gb/sec Ethernet. The server adapter cards also support Remote Direct Memory Access over Converged Ethernet, or RoCE, which provides low latency on Layer 2 networks that rivals InfiniBand.

Like existing cards, the Pro cards support Windows, Linux, and FreeBSD operating systems and VMware ESXi and Citrix Systems XenServer hypervisors. (Presumably Hyper-V support is on the way.)

Mellanox is sampling the new cards now and will ship them in May. Pricing will be announced at that time. ®