Arista monster switches fluff up cloud with 1 million virty machines

7500E mod box sports 100Gb/sec speeds and integrated optics on ports

Next gen security for virtualised datacentres

Data center switch vendor Arista Networks is giving the incumbent peddlers of switchery heartburn again with the launch of its 7500 E Series modular switches.

The upstart company, which has Sun Microsystems co-founder Andy Bechtolsheim as its chairman and chief development officer, is taking switching up another notch on a few different dimensions with the 7500 E Series. First, the two 7500 E switches launched today are based on the BCM88650 ASIC from Broadcom, which comes out of the Dune Networks part of the house. (Broadcom acquired Dune Networks back in November 2009 for $178m.) The 7500E uses six BCM88650 chips per linecard.

The BCM88650 is the fastest and fattest switch ASIC that Broadcom makes and allows for up to eight modular switches based upon it to be linked together without a fabric and up to 32 to be lashed into one domain if you use the fabric element coprocessor, the BCM88750. This ASIC delivers 200Gb/sec at full duplex and can push it up to 240Gb/sec if you do a little oversubscribing on the ports.

It looks like Arista has a slightly goosed version of the BCM88650 chip in that it can push 30Tb/sec of bandwidth across its ports in a full fabric configuration while the stock chip from Broadcom does 25Tb/sec according to its data sheet (PDF). The resulting fabric that is created from six BCM88650 ASICs on a line card and eight linecards in the 7500 E Series chassis is a switch that can handle 14.4 billion packets per second of traffic. You are talking 460.8 billion packets per second for this behemoth.

The BCM88650 also supports VMware's VXLAN Layer 2 overlay method for Layer 3 networks, which allows for virtual machines to flit across distributed data centers or pods not on the same Layer 2 net without breaking their network links.

Arista has two new 7500E modular switches

Arista has two new 7500E modular switches

The second important aspect of the 7500E switches is that the ports on the switch can be programmatically dialed, through the EOS network operating system that Arista created for its switches, to run at 10Gb/sec, 40Gb/sec, or 100Gb/sec speeds, and that means Arista can make a bunch of different line cards available relatively easily.

Interestingly, the transceivers behind the ports of the switch have integrated the optics inside the chassis and on each port that would normally be embedded in optical cables. This means you do not have to spend big bucks on revved-up optical cables to support 100Gb/sec speeds, Anshul Sadana, vice president of customer and systems engineering at Arista, explains to El Reg, and the integration at the switch means you can cut the cost of deploying 100Gb/sec Ethernet networks by a factor of ten.

There are two different models of the 7500E modular switch from Arista. The DCS-7504 has four linecard slots in a 7U enclosure, while the DCS-7508 has eight linecard slots in an 11U chassis. Both machines have six fabric module slots, with each fabric module sporting either 2.6Tb/sec or 5.12Tb/sec of bandwidth.

That yields 15.35TB/sec of bandwidth and 7.2 billion packets per second of oomph for the four-slot switch and 30.7Tb/sec of bandwidth and 14.4 billion packets per second processed. The four-card switch can have 40 ports running at 100Gb/sec, 144 ports running at 40Gb/sec, or 576 ports running at 10Gb/sec, while the eight-slotter obviously doubles all of this up. Sadana says you can do a port-to-port hop on one of these babies in something like 4 to 5 microseconds.

The 7500e has four different line cards to suit various needs

The 7500E has four different line cards to suit various needs

Different folks will need a different mix of ports, of course, so Arista has welded together the Broadcom ASICs and various port configurations into four different linecards. If you want to go full-tilt-boogie on 100Gb/sec, there is a line card that has a dozen 100Gb/sec ports with the integrated SR10 optics on the transceivers on the linecard.

Those ports can be dialed back to 10Gb/sec or 40Gb/sec ports if you are not ready to run at 100Gb/sec, and importantly has an 18GB port buffer that works in conjunction with the 144GB switch packet buffer (three times that in the older 7500 modular switch). The second card has 36 ports running at 40Gb/sec speeds with QSFP+ ports and matching

cables, which have their own optics; it also has an 18GB port buffer. The third line card has 48 ports using SFP+ cables and running at 40Gb/sec speeds (that can be dialed back 1Gb/sec or 10Gb/sec speeds and, in the case of 10Gb/sec, can use splitter cables to turn one port into four; this card also has two 100GB/sec ports, which can be used as uplinks and has a 9GB port buffer. And finally, the fourth line card has just 48 ports with SFP+ connectors with a 9GB port buffer.

Arista loves scalability and bandwidth, of course, but at 4 watts per port, Sadana says that the 7500E is three times as energy efficient as the 7500 switch it replaces. And Arista is very proud about the pricing it can deliver for 100Gb/sec ports.

If you take a fully loaded 7500E switch and load it up with 10Gb/sec SFP+ line cards and hang optical cables off it, you pay $920 per port for the switch and $450 per port for the optical cables, or $1,370 per 10Gb/sec per port.

If you do the same math for a 40Gb/sec line card with QSFP+ optical cables, you are in for $3,000 per port for the switch and $2,000 per optics cable, but if you adjust that down by bandwidth, it actually works out to $1,250 per 10Gb/sec segment of bandwidth per port. And on 100Gb/sec switches, which have the integrated optics, the cost of a 10Gb/sec slice on a port is $1,220, even lower. (Yes, that is still $12,200 per port including the 100m multimode cable.)

How does more than 100,000 servers on a single network grab you?

How does more than 100,000 servers on a single network grab you?

Of course, what Arista is after are customers who need to build very flat Layer 2/3 networks and who want to spend a lot less money on them than they are doing with Cisco Systems. Sadana says that it can build a leaf/spine network with 6,912 ports using two 7500E modular switches and 144 of its 7150 top-of-rackers.

It takes eighteen Nexus 7000 core and aggregation switches a 144 Nexus 5000 top-of-rack switches from Cisco to do the same number of ports. (This comparison assumes 3:1 oversubscription on the core for both networks.) The Arista gear takes up half the rack space, has eight times lower energy consumption per port, and one-fifth the latency. Cisco will no doubt say that Arista has cooked up a flawed comparison.

The 7500 E Series modular switches are piloting now, and will be generally available in the second quarter. The base 7500E is expected to cost $99,995 and work out to around $600 per 10Gb/sec port, $2,200 per 40Gb/sec port, and around $10,000 per 100Gb/sec port at the switch level. This price does not include cables as the comparison above did. ®

Build a business case: developing custom apps


Best practices for enterprise data
Discussing how technology providers have innovated in order to solve new challenges, creating a new framework for enterprise data.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Advanced data protection for your virtualized environments
Find a natural fit for optimizing protection for the often resource-constrained data protection process found in virtual environments.
How modern custom applications can spur business growth
Learn how to create, deploy and manage custom applications without consuming or expanding the need for scarce, expensive IT resources.
High Performance for All
While HPC is not new, it has traditionally been seen as a specialist area – is it now geared up to meet more mainstream requirements?