Sun men bump bellies in Cisco v Arista freaky trade battle
'You call that a low-latency switch, buddy?'
High frequency trading and certain parallel computing tasks need low-latency switches with as little jitter as possible, and they are willing to pay a hefty premium for those qualities. This was one of the founding premises of switch upstart Arista Networks, and Cisco Systems has revved up its Nexus 3000 series switches to try to carve out a piece of this profitable market from its upstart rival.
The battle between Cisco, Arista, Juniper Networks, and a handful of other rivals that have not been acquired by an incumbent server maker is pitting Andy Bechtolsheim, co-founder of Sun Microsystems, Granite Systems, Kealia, and now Arista against David Yen, a former top exec in charge of chips, servers, and storage at various times in his career at Sun Microsystems who spent a few years at Juniper before landing at Cisco in May 2011.
Yen left Sun Microsystems for networking upstart Juniper Networks back in March 2008, a year before Arista came out of stealth mode and just as it was beginning to ship alpha versions of its goosed 10 Gigabit Ethernet switches based on ASICs from Fulcrum Microsystems (now part of Intel) and Broadcom.
At Juniper, Yen was executive vice president of emerging technology, a rather vague position, but in the jump to Cisco he was named general manager of the Data Center Technology Group at Cisco - and that puts him in charge of data center switching (stable but not growing), servers (growing like crazy), storage (Cisco is still in the storage biz?), and load balancing (just dumped that biz unit this week).
Bechtolsheim is no stranger to Cisco, either, and ricocheted in and out of Sun a few times. And frankly, the big surprise is that Cisco was not smart enough to buy Arista several years ago. With $48bn in the bank, it was certainly rich enough to do so with pocket change.
Bechtolsheim was CTO at Sun until 1995, when he left to commercialize Gigabit Ethernet switches with a company called Granite Systems. Cisco bought Granite for $220m in stock in September 1996, and this is the basis of the Catalyst 4000 series of switches at Cisco. Bechtolsheim stuck around Cisco for a few years, and then caught the InfiniBand networking bug in February 2001: founding a startup called Kealia, which meshed InfiniBand switches with dense-packed servers and storage to create the "Constellation" system. You recognize this machinery, because Sun Microsystems bought Kealia for an undisclosed amount in February 2004 and it was the basis of Sun's resurgent server, storage, and networking business. It survives at the heart of Oracle's Exadata and Exalogic clusters to this day. Bechtolsheim copped to founding Arista in late 2008, and stuck around as Sun's server CTO for a few months.
If the Bechtolsheim pattern had continued, Cisco would have bought Arista two years ago and Bechtolsheim would be working at Oracle, thinking of the next networking biz to start. Which is why you have to be careful about seeing patterns in the world.
The one thing that Cisco and Arista both know is that Arista has carved out a nice little niche for itself in the networking racket among high freaky traders and supercomputer boffins. Arista tells El Reg that it estimates that it has somewhere around 70 to 80 per cent market share in the low-latency Ethernet switch space – the kind of market share that Cisco likes to command because it drives margins.
So Cisco is taking on Arista's 7100 series of machines, which were just updated this week, with the Nexus 3548. Unlike Arista, Cisco is using its own electronics inside the box, and is cooking in a feature called Algo Boost to crank up the speed of algorithms commonly used in high frequency trading.
The Nexus 3548 high freaky trading switch
Cisco's prior Nexus 3000 series switches, which were launched in October 2011, were based on ASICs designed by others (presumably Intel/Fulcrum and Broadcom, but Cisco doesn't talk about that). The Nexus 3064 came out in March 2011 specifically for speed freaks and resembled the Arista 7050S-64 that was announced the same week.
The Nexus 3548 is a 1U rack switch that sports 48 10GE ports. The Cisco ASIC, code-named "Monticello," has 960Gb/sec of aggregate switching bandwidth, and has a forwarding rate of 720 million packets per second. That is not as much bandwidth or forwarding as Arista is delivering with its new 7150S switch, which is based on Intel's "Alta" FM6000 ASIC and delivers 1.28Tb/sec of switching bandwidth and can handle 960 million packets per second forwarding. Depending on how many ports it is driving, the 7150S ranges between 350 and 390 nanoseconds on a port-to-port hop inside the switch, and that is a lot better than 500 nanoseconds that prior Arista boxes aimed at high freaky traders could do.
The only trouble for Arista is that Cisco has normal, warp, and warp SPAN modes that radically reduce the latency on port-to-port hops inside the Nexus 3548. In normal mode where you can attach many devices to the switch with complex Layer 2 and 3 forwarding tables, you can get as low as 250 nanoseconds of latency with the Nexus 3548. In warp mode, the ASIC consolidates forwarding operations off other parts of the switch onto the ASIC and reduces the number of MAC addresses, IPv4 unicast and multicast routes, and IPv4 hosts significantly and thereby boosts the latency by another 20 per cent or so to around 190 nanoseconds. In warp SPAN mode (short for Switch Port Analyzer), where switches are really just taking in market data streams and forwarding them to one port or many ports at the same time, the Monticello ASIC can do this with as little as 50 nanoseconds of latency.
This multi-speed tranny in the Nexus 3548 is part of the Algo Boost functionality of the ASIC, which also has other features to boost performance of algorithms running on data streams and trading transactions. This includes network address translation (a feature required by IP networks in trading systems, and also added by Arista to the new 7150S switch) without any latency delays. Active buffer monitoring allows for the Monticello ASIC to sample its buffer utilization data in 10 nanosecond timeslices and thereby become more aware of congestion before traffic gets backed up too far on any particular port. Advanced traffic monitoring is another Algo Boost feature that can filter data as it burns through the switch as well as timestamp it, features that are necessary to reduce the latency on financial transactions and to guarantee the right order of transactions in those trading systems.
Like other Nexus switches, the 3548 runs the NX-OS network operating system. Interestingly, the switch has no physical layer (PHY) chips, which is one of the ways Cisco was able to cut the latency and power consumption of the unit. The Nexus 3548 also has a port for a grandmaster timing clock that adheres to the IEEE-1588 Precision Time Protocol (PTP) standard for synchronizing switches.
The Nexus 3548 is available now and has a base list price of $41,000, or around $854 per port. That's considerably more expensive than the 24-port Arista 7150S switch, which costs $12,995 or around $541 per port. As you might expect in high freaky trading, time is after all money. ®