Mellanox stretches InfiniBand across campuses, metro areas

Adds baby FDR switch, grafts Layer 3 onto Ethernet switches

SC12 With InfiniBand on the march, taking market share away from Ethernet in the upper echelons of the supercomputer racket, Mellanox Technology could sit back and enjoy itself a little. But instead, the company is tweaking its InfiniBand lineup to make its switches and adapters more appealing and to keep InfiniBand rolling.

The networking company is at the SC12 supercomputing conference this week, talking up its new wares, which includes long-haul extensions to InfiniBand that will allow it to span campuses today and metro regions at some time in the not-too-distant future.

Mellanox is also pumping out a low-end InfiniBand switch with only a dozen ports (suitable for more modest uses), bragging about the performance of its Connect-IB server adapter cards, adding Layer 3 routing to its Ethernet switches, and forging a hardware appliance to run its Unified Fabric Manager fabric management software.

The first of the new MetroX series of long-distance InfiniBand switches is called the TX6100 (yes, I know, why is it not called the MX6100?), and it is able to drive 10Gb/sec over dark fiber over distances as large as 10 kilometers. If you want to drive at higher 40Gb/sec bandwidths, then you can only push as far as 2 kilometers with the TX6100.

The TX6100 has six long-haul ports and six downlinks rated at the 56Gb/sec FDR InfiniBand speeds. The switch has a port-to-port latency of 200 nanoseconds and a latency of around five nanoseconds per meter over the dark fiber. (Mellanox is clever, but it has not been able to break the light-speed barrier yet.) The TX6100 is 1U high, has a switching capacity of 240Gb/sec, and burns around 200 watts.

In the third quarter of this year, the MetroX-2 TX6200 long-haul InfiniBand switch will debut, and it will have essentially the same feeds and speeds but will be able to drive 40Gb/sec over dark fiber at a distance of 10 kilometers.

The MetroX TX6000 long-haul InfiniBand switch

The MetroX TX6000 long-haul InfiniBand switch

Further down the road, Mellanox is promising customers that with the MetroX-3 series of long-haul InfiniBand switches will be able to drive 40Gb/sec over 100 kilometers.

With InfiniBand being increasingly used to link servers to each other in high-performance clusters and becoming the default high-speed link for storage clusters and to link said clusters to similar server clusters, the need for some kind of native InfiniBand long haul switch was acute, Gilad Shainer, vice president of market development at Mellanox, tells El Reg.

Companies, universities, and government labs want to have long-haul links between their storage arrays to replicate data, and they want to link server clusters together over campuses, too. The financial services industry, particularly in the wake of Hurricane Sandy in New York, wants to have mirroring to remote disaster recovery sites and be able to failover.

And, in the story we have heard for more than three decades, long-haul InfiniBand would allow for big, fat patient data to be shared between hospitals and doctors in a metro area.

While there were ways to convert InfiniBand to IP protocols and beam data out over dark fiber (which is not fiber that is not lit up, but rather the industry name for leased telecom fiber capacity in a metropolitan area), these protocol conversions smashed one of the key features of InfiniBand that give it low latency: the Remote Direct Memory Access (RDMA) protocol.

With the MetroX switches, RDMA works over the long haul, allowing for InfiniBand adapters to reach out over the network locally and now remotely to grab data right out of the main memory of a server anywhere on the LAN or WAN.

Although Mellanox did not say this, the TX6100 is almost certainly based on its new the new SwitchX-2 ASIC launched Mellanox last month, which has a number of features for software-defined networks (SDN) etched into them. One of them was called routable RDMA, which allowed RDMA requests to be sent over Layer 3 of the network.

The TX6100 is being demonstrated at SC12 this week and will ship before the end of the year. Pricing was not available since the product is not yet shipping.

Two small switches are sometimes better than one big one

The smallest InfiniBand FDR switch that Mellanox has been selling has 36 ports, and for a number of different uses, this machine is too big for the job. You might have a small departmental cluster for application development or a modest production workload such as a small database or HPC cluster. Or, in some cases, you want to build a storage cluster with a modest number of nodes and you don't want to pay for switching capacity you are not going to use.

And so Mellanox is kicking out the SX6012 and the SX6005 switches, which sport twelve ports. Both machines are based on the SwitchX-2 chips, with the SX6012 being an internally managed switch and the SX6005 being an externally managed variant.

The baby sx6012 InfiniBand switch

The baby sx6012 InfiniBand switch

These switches support ports running up to the top-end 56Gb/sec FDR speeds, have 1.34Tb/sec of aggregate switching capacity, and a port-to-port latency hop of 170 nanoseconds.

They are half-width 1U switches that have an adapter kit to mount two of them side-by-side in a rack, which is going to be useful for storage clusters that need redundant switches. The SX6012 and SX6005 switches are available now, but Mellanox did not provide pricing at press time.

Further on the hardware front, Mellanox has been told by customers that they are sick of installing its Unified Fabric Manager on a server of their own and they just want to buy an appliance that is preconfigured with the network control freak.

The UFM-SDN appliance from Mellanox: Plug and go

The UFM-SDN appliance from Mellanox: Plug and go

The UFN-SDN appliance – everything from every networking vendor will now be SDN, just like every server will be cloud – has two Xeon E5-2630 processors running at 2.3GHz with 32GB of memory, two 1TB 7200 RPM disks that are mirrored, and two 750 watt power supplies.

The UFM 4.0 software installed on the appliance has SDN capability that was built into the InfiniBand from the get-go (called Subnet Manager), and for the moment it hooks into OpenFlow controllers for Ethernet. Shainer tells El Reg that Mellanox will extend UFM with OpenFlow controller functions in the long run, and was as vague about when and how as he was back in October when the SwitchX-2 ASICs debuted.

Mellanox is also bragging about its two-port Connect-IB InfiniBand server adapter card, which launched at the International Super Computing event in June. The Connect-IB card has two 56Gb/sec FDR ports and plugs into a PCI-Express 3.0 slot, which at this point means a server based on a Xeon E3 or E5 processor since no other kind of machine sports a 3.0 slot right now.

The two-port adapter has been tested to drive over 100Gb/sec out of a single x16 slot, processing over 135 million messages. That is 4.5X more messages than its prior PCI-Express 2.0 ConnectX-2 cards could manage to do.

There are other Connect-IB variants with a single port that slide into a PCI-Express slot and a slightly less impressive x8 one for PCI-Express 3.0 slots that can drive two ports, but just not at that rate. The Connect-IB boards are sampling now, and the shipment date was not released.

Finally, Mellanox says that its MLNX-OS switch operating system has been tweaked with networking features to allow routing over Layer 3 in the network. This functionality is enabled in any SwitchX or SwitchX-2 switch that is set up to drive 10 or 40 Gigabit Ethernet.

The VXLAN standard for virtual LAN overlays from VMware and the competing NVGRE standard from Microsoft both ride atop the Layer 3 support that Mellanox has added to MLNX-OS. This updated feature set will be patched onto MLNX-OS before the end of the year, according to Shainer. ®

Sponsored: Minds Mastering Machines - Call for papers now open

Biting the hand that feeds IT © 1998–2018