InfiniBand to outpace Ethernet's unstoppable force
Run faster or be crushed
SaaS data loss: The problem you didn’t know you had
Comment Every good idea in networking eventually seems to be borged into the Ethernet protocol. Even so, there's still a place in the market for its main rival in the data center, InfiniBand, which has consistently offered more bandwidth, lower latency, and often lower power consumption and cost-per-port than Ethernet.
But can InfiniBand keep outrunning the tank that is Ethernet? The members of the InfiniBand Trade Association, the consortium that manages the InfiniBand specification, think so.
InfiniBand, which is the result of the merger in 1999 of the Future I/O spec espoused by Compaq, IBM, and Hewlett-Packard and the Next Generation I/O competing spec from Intel, Microsoft, and Sun Microsystems, represents one of those rare moments when key players came together to create a new technology — then kept moving it forward. Sure, InfiniBand was relegated to a role in high-performance computing clusters, lashing nodes together, rather than becoming a universal fabric for server, storage, and peripheral connectivity. Roadmaps don't always pan out.
But since the first 10Gb/sec InfiniBand products hit the market in 2001, it's InfiniBand, more than Ethernet, that has kept pace with the exploding core counts in servers and massive storage arrays to feed them, which demand massive amounts of I/O bandwidth in the switches that link them. Which is why InfiniBand has persisted despite the onslaught of Ethernet, which jumped to Gigabit and then 10 Gigabit speeds while InfiniBand evolved to 40Gb/sec.
Now the race between InfiniBand and Ethernet begins anew. As El Reg previously reported, the IEEE has just ratified the 802.3ba 40Gb/sec and 100Gb/sec Ethernet standards, and network equipment vendors are already monkeying around with non-standard 100Gb/sec devices. At the SC09 supercomputing conference last fall, Mellanox was ganging up three quad data rate (QDR, at 40Gb/sec) InfiniBand pipes to make a twelve-port 120Gb/sec switch. This latter box is interesting, but it is not adhering to the current InfiniBand roadmap:
InfiniBand is a multi-lane protocol. Generally speaking, says Brian Sparks, co-chair of the IBTA's marketing working group and the senior director of marketing at Mellanox, the four-lane (4x) products are used to link servers to switches, the eight-lane (8x) products are used for switch uplinks, and the twelve-lane (12x) products are used for switch-to-switch links. The single-lane (1x) products are intended to run the InfiniBand protocol over wide area networks.
As each new generation of InfiniBand comes out, the lanes get faster. The original InfiniBand ran each lane at 2.5Gb/sec, double data rate (DDR) pushed it up to 5Gb/sec, and the current QDR products push it up to 10Gb/sec per lane.
Next page: Life in the faster lane
COMMENTS
1 gbit ether seems quite good with Hadoop
This article seems more like a press release from the infiniband marketing organisation than anything else.
1. Hadoop and MapReduce doesn't need 10 gbit/s to the servers, 1 gigabit is fine for all but the backbone.
2. Facebook have just been describing how they use bittorrent to push out OS images as it scales well on gigabit.
3. 10 gbit/s has a worse power budget and is tricker to keep alive, though optical cabling can save on electricity
When you consider that all the big datacentres appear to be going for JBOD storage on the servers with gigabit interconnect, it's hard to say the trend towards SAN mounted storage and inifiniband is unstoppable.
CSMA/CD? 4000MTU?
In an Infiniband discussion you guys confuse current Ethernet!
There is no CSMA/CD in any Point-to-Point full duplex or switched Ethernet, nor for over 5 years - so it has been gone in most deployments for a long time, and was last seen with hubs.
Jumbo frames - part of the 1GB spec which is also ages old have been approx 8000 for years now, but again many shops do not implement.
The big difference with infiniband, is that you can work within the seven layer ISO model, and switch the Ethernet and infiniband electrical layers quite easily at a technical level, it is cost (largely infrastructure investment) that works against that.
There is also a large body of people who confuse ethernet (lower layers) with the TCP/IP related higher layers which are quite portable.
CSMA/CD is so last week
It's not in use on full-duplex links, and while theoretically still possible, there's not a single commercial gigabit ethernet adapter that'll run half-duplex in existence. Beyond that? Don't be silly.
To me ethernet always had the big advantage of being far simpler than its main competitors, but that is its greatest weakness as well. An MTU of 1500 was a cost-reducing measure, and while a reasonable tradeoff then, it's much less so now. And we have plenty trouble shaking it off. Also: QoS. Instead of offering support (like VG did), ethernet requires layers above it to reinvent it (badly).
While without cheap networking the world would look... different, we are being hampered by the limitations of our previous tradeoffs. For a thought experiment, suppose we would have had 10Gigabit Arcnet now. Discuss.
IT infrastructure monitoring strategies
Agentless Backup is Not a Myth
Top 10 SIEM implementer’s checklist
Steps to Take Before Choosing a Business Continuity Partner
Enabling efficient data center monitoring
