Original URL: http://www.theregister.co.uk/2009/06/05/cray_swiss_xt5/
Cray sells Opteron-Linux super to Swiss boffins
Monte Rosa to scale the petaflops peaks
The Swiss National Supercomputing Center (CSCS) in Lugano has coughed up some serious Swiss francs to buy a Cray 141.6 teraflops XT5 system, dubbed Monte Rosa.
The deal comes in part from the fact that Thomas Schulthess, director of CSCS and a professor of physics at ETH Zurich, took over the lab after a stint at Oak Ridge National Laboratory, where the U.S. government has paid for the "Jaguar" massively parallel 1 petaflops Opteron-Linux XT5 supercomputer.
Call it petaflops envy. CSCS has been buying Cray machines for some time, and this is really an upgrade to an existing - and much less powerful - Cray box.
The initial configuration of Monte Rosa will consist of 3,688 Advanced Micro Devices quad-core Shanghai Opterons - that's 14,752 total cores - spinning at 2.4GHz and backed up by 29TB of main memory spread across the XT5 processor nodes in the parallel system.
CSCS says that in November of this year it will swap out the Shanghai chips and upgrade the XT5 box to use the new six-core Istanbul Opterons announced this week, for a total of 22,128 cores running at the same 2.4GHz and delivering 212 teraflops of number-crunching power. That Istanbul setup of Monte Rosa will have a total of 59TB of main memory across its nodes.
The CSCS facility in Lugano was the first site in Europe to take delivery of Cray's XT3 systems back in 2005, and the Swiss government has just put funding in place to get the CSCS up to the petaflops level with its supers by 2012.
The XT3 is the commercialized version of the "Red Storm" parallel supercomputer that Cray built for the U.S. government's Sandia National Laboratories. Commercial shipments of the XT3 began in 2005, and in 2006 the boxes were upgraded to dual-core Opterons.
The XT4 machines were announced at the end of 2006 using dual-core Opteron 1000 series processors (not the 2000 series chips used in standard two-socket servers), and were to be upgraded to four-core Opteron 1000 series chips when AMD got them out the door around the middle of 2007.
The XT4s were designed to support 30,000 dual-core Opteron processors, but AMD was tragically late delivering those Budapest parts. By November 2007, Cray was already launching the XT5 supers, which were designed to support standard Opteron 2000 parts.
The XT5 supers use a blade form factor for server nodes, with eight Opterons, lots of main memory, and four SeaStar2+ torus interconnect chips to link the Opteron HyperTransport buses to the world outside of the blade.
The XT5 is pretty dense compared to the XT3. About 20 times as dense, in fact, so the Monte Rosa super fits in 20 racks at 141.6 teraflops compared to 120 racks for the Red Storm machine rated at 43 teraflops.
Exactly how much money has been budgeted to get to petaflops performance is not clear, because that information is only available in Swiss German. (Lugano is the southern area of Switzerland that speaks Italian, so go figure.) If you trust Yahoo!'s Babelfish translation, it appear that the Swiss Bundesrat fördert Hochleistungsrechnen is ponying up 172.5 million francs (£99.5m or $159m) to build up the Swiss supercomputer complex, which will be spent in stages.
It's unclear if the petaflops super the Swiss government envisions will use AMD's future Opteron or Intel's future Xeon processors, since Cray itself has not said publicly what it plans to do beyond the current XT5 designs. But after the quad-core Budapest delay - which really cost Cray financially - it would not be surprising to see Cray make a system that could take either Xeon or Opteron processors. That task would be made much easier by Cray's partnership with Intel, announced a year ago when the Barcelona quad-core Opterons were just getting debugged, and by the fact that the Opteron's HyperTransport and the Nehalem Xeon's QuickPath Interconnect are very similar. ®