Feeds

Mellanox pushes InfiniBand to 120Gb/s

Offloading IB processing from CPUs to ConnectX-2s

HP ProLiant Gen8: Integrated lifecycle automation

SC09 If 40 Gb/sec InfiniBand is not enough for you, then you'll be happy to hear that InfiniBand switch maker Mellanox Technologies is going to crank its switches past 11 to deliver 120 Gb/sec ports in its MTS and IS switch families.

As it turns out, the InfiniScale IV chips that Mellanox created for its 40 Gb/sec IS5000 switches, which were announced in June, are able to support 120Gb/s ports. The company said at the SC09 supercomputing trade show in Portland, Oregon, that in the first quarter of 2010, it will ship a variant of the MTS3600 fixed-port switch that employs CXP connections, which are already used with 40 Gb/sec InfiniBand by a number of vendors, that gang up 12 lanes of 10 Gb/sec InfiniBand traffic into a single port.

The MTS3600 comes in a 1U box that supports 36 40Gb/s ports, and a variant of this box supporting CXP links will come out with 12 ports running at 120Gb/s. The leaf modules in the IS5000 series of modular switches, which came out in June as well, have their port counts cut by a third and their bandwidth per port tripled.

Mellanox was demonstrating the new 120 Gb/sec switches at the show, and the high-speed InfiniBand switches were the backbone of a 400 Gb/sec network supporting the show that SCInet, the network provider for the SC trade shows, slapped together and has a value of $20m if you had to buy it. (The SC event does not have to buy its network, since vendors are thrilled to donate equipment and experts to be part of the high-speed backbone.)

Eventually, the 120 Gb/sec product line will include fixed-port switches with 12, 36, and 72 ports and hybrid 120Gb/sec and 40 Gb/sec switches that have six of the fast ports and 18 of the slower ones.

According to John Monson, vice president of marketing at Mellanox, the price per bit on the 120 Gb/sec variants of the switches will be the same as on the 40 Gb/sec, but the port costs will obviously triple. The reason why HPC shops would want to move to 120 Gb/sec is simple: by ganging up the InfiniBand pipes, users can cut the InfiniBand transport overhead by a factor of three and squeeze more performance out of their clusters. The switches will also allow for more bandwidth to be allocated in 3D torus systems between the cubes that make up the torus.

Enabling the upgrade to 120 Gb/sec is a "golden screwdriver" patch to the firmware, since the InfiniScale IV chips already supported the higher bandwidth, but obviously the CXP ports and their optical cables are different and you have to buy a new box to get them.

Mellanox also said at the SC09 show that the ConnectX-2 host channel adapters it has been shipping had another golden screwdriver upgrade. The prior generation of ConnectX IB cards allowed for some of the processing related to the InfiniBand protocol to be offloaded from CPUs in the server nodes to the IB cards. (Much as TCP/IP has had offloading for Ethernet cards for a number of years now.)

With the ConnectX-2 cards, Monson says that Mellanox has gone one step further and put electronics in the HCA that can take over some of the Message Passing Interface (MPI) collectives operations - those that broadcast data, gather data, or otherwise synchronize the nodes in a cluster.

Mellanox has done benchmark tests that show that clusters lose 20 to 30 per cent of their scalability from this MPI collectives communications, and that means cycles on the CPUs that could be doing other work are stuck doing this MPI work. Hence the offload, which is turned on with a firmware change on the existing cards and which cuts down on jitter and noise in the cluster and lets it get more work done.

Interestingly, the ConnectX-2 IB cards also have an embedded floating point co-processor, which can take over some of the calculation jobs that the MPI stack sends to a server node in the cluster, provided it has a Mellanox IB card.

Mellanox has been working with Oak Ridge and Los Alamos, two of the national laboratories funded by the US Department of Energy, to tweak the MPI stack to see this embedded floating point unit. Monson is mum about when this capability will ship be activated. ®

Reducing security risks from open source software

More from The Register

next story
Sysadmin Day 2014: Quick, there's still time to get the beers in
He walked over the broken glass, killed the thugs... and er... reconnected the cables*
SHOCK and AWS: The fall of Amazon's deflationary cloud
Just as Jeff Bezos did to books and CDs, Amazon's rivals are now doing to it
Amazon Reveals One Weird Trick: A Loss On Almost $20bn In Sales
Investors really hate it: Share price plunge as growth SLOWS in key AWS division
US judge: YES, cops or feds so can slurp an ENTIRE Gmail account
Crooks don't have folders labelled 'drug records', opines NY beak
Auntie remains MYSTIFIED by that weekend BBC iPlayer and website outage
Still doing 'forensics' on the caching layer – Beeb digi wonk
BlackBerry: Toss the server, mate... BES is in the CLOUD now
BlackBerry Enterprise Services takes aim at SMEs - but there's a catch
The triumph of VVOL: Everyone's jumping into bed with VMware
'Bandwagon'? Yes, we're on it and so what, say big dogs
Carbon tax repeal won't see data centre operators cut prices
Rackspace says electricity isn't a major cost, Equinix promises 'no levy'
prev story

Whitepapers

Designing a Defense for Mobile Applications
Learn about the various considerations for defending mobile applications - from the application architecture itself to the myriad testing technologies.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Top 8 considerations to enable and simplify mobility
In this whitepaper learn how to successfully add mobile capabilities simply and cost effectively.
Seven Steps to Software Security
Seven practical steps you can begin to take today to secure your applications and prevent the damages a successful cyber-attack can cause.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.