Original URL: https://www.theregister.com/2011/01/03/prace_lrz_supermuc_super/

Germans pick Xeon over Power for three-petaflops super

Big Blue licks Bavarian HPC cream

By Timothy Prickett Morgan

Posted in HPC, 3rd January 2011 21:56 GMT

The Partnership for Advanced Computing in Europe (PRACE), which is upgrading its pan-European supercomputing facilities across a number of large HPC labs, has announced that it has commissioned IBM to build a 3-petaflops massively parallel super using the company's iDataPlex blade-rack hybrid machines and the next-generation of Intel Xeon processors.

This machine, known as SuperMUC, is to be installed at the Leibniz-Rechenzentrum (LRZ), which is home to a big Itanium-based Altix 4700 cluster made by Silicon Graphics. SuperMUC will be a companion to the 1-petaflops "Jugene" BlueGene/P massively parallel box that the Forschungszentrum Juelich (FZJ) in North Rhine-Westphalia paid IBM for back in February 2009.

The other PRACE big box is the "Curie" massively parallel Xeon 7500-based super made by Bull for the Commissariat a l'Energie Atomique (CEA) outside Paris. The Jugene machine has 294,912 of IBM's 850MHz PowerPC cores, and has a peak performance of just over 1 petaflops, with sustained performance on the Linpack Fortran benchmark test of 825.5 teraflops. The current CEA super in France, known as the Tera 100, has 138,368 of Intel's Xeon 7500 cores, yielding 1.25 petaflops of peak and 1.05 petaflops of sustained performance on the Linpack test. The Curie super at CEA is being built this year and has 360 nodes using current Xeon 7500 processors from Intel, rated at around 100 teraflops aggregate, with a 1.5 petaflops cluster based on future Xeons coming later this year.

The LRZ, which is located in Garching, Germany, did not divulge all of the feeds and speeds of the forthcoming SuperMUC machine, but did say it would be built from the iDataPlex tray servers in the System X product line, and not from standard System x rack servers or BladeCenter blade servers from IBM.

The LRZ added that SuperMUC will be up and running by the middle of 2012, would be rated at a peak 3 petaflops of number-crunching power, would have an aggregate of 320TB across its server nodes, and would be fed by 12PB of storage. The lab said further that the machine would use more than 110,000 cores to reach that 3-petaflops performance level.

In the picture below, Arndt Bode, chairman of LRZ (which is part of the Bavarian Academy of Sciences and Humanities), and Martin Jetter, chairman of IBM Germany, are holding up one of the tray servers from the future SuperMUC machine:

SuperMUC Supercomputer Server Node

You can't see a lot from this picture, of course, but it looks like it's a two-socket tray server. There's a lot of speculation that the server nodes in SuperMUC will be based on the forthcoming "Sandy Bridge" Xeon processors, which Intel is cooking up in a 32-nanometer process.

Server versions of the Sandy Bridge chips are expected in the third quarter of 2011 for EP variants for two-socket boxes with relatively high clock speeds and low numbers of memory slots, and maybe in the fourth quarter for EX variants for two-socket, four-socket, and eight-socket boxes, generally with lower clock speeds but much more main memory per socket. Intel is getting ready to launch the PC versions of the Sandy Bridge chips this week at the Consumer Electronics Show in Las Vegas.

It seems highly unlikely that SuperMUC would be based on the ten-core "Westmere-EX" chip due in the second quarter of this year, presumably to be called the Xeon 7600 and socket-compatible with the eight-core Xeon 7500 processors. If LRZ had chosen this 32nm chip, it could have its box up and running probably in the third quarter of this year, not the middle of 2012.

As El Reg explained back in June, an eight-core Sandy Bridge Xeon chip for servers running at 3GHz would be able to do eight double-precision flops per clock, or 192 gigaflops per socket. PRACE said the SuperMUC machine would have more than 14,000 processors and more than 110,000 cores, so we know they are using a Xeon chip with eight cores. At 3.3GHz, that's 211 gigaflops per socket. If you have 84 racks of iDataPlex machines, which each have 84 servers with two cores each, that's 14,112 sockets and 2.98 petaflops of peak performance. That sure fits the data pretty well.

But there is a decent chance that LRZ might skip a generation and go straight to the "Ivy Bridge" 22-nanometer shrink of the Sandy Bridge Xeon server chips. It is expected that both Sandy Bridge and Ivy Bridge will share the LGA 2011 socket, so the iDataPlex platform would be the same either way. But with a 22nm shrink to Ivy Bridge, the chips would run a lot cooler for an eight-core version compared to Sandy Bridge, and heat is a big deal for HPC centers.

If Intel can get Ivy Bridge kickers to LRZ early – much as CEA got a slew of early Xeon 7500s to build another of the PRACE machines last year with Bull – then there's an outside possibility that the SuperMUC machine could use a 3.3GHz Ivy Bridge part and get the same performance, possibly pushing down from a 130-watt Sandy Bridge part to a 95-watt or even an 80-watt Ivy Bridge part – those are my guesses for the kind of energy savings you might see in the shrink from 32nm to 22nm, and saving a load of energy consumption and heat.

What IBM is saying for sure is that SuperMUC will have water blocks on processors and main memory on the iDataPlex system boards, and a closed-loop water-cooling system that will use relatively warm water (up to 45 degrees Celsius, which is 113 degrees Fahrenheit) to keep these active components from melting. IBM installed a similar cooling system on a Cell-based baby super called Aquasar at the Eidgenössische Technische Hochschule back in July 2010, which can cool the machine with water that is as hot as 65 degrees Celsius.

The future BlueGene/Q massively parallel super, which El Reg told you about here, and the Power7 IH supercomputer node that is part of the "Blue Waters" parallel super, which we told you about here, both have direct water-cooling on the CPUs and memory. The Power7 IH node used in Blue Waters, which is being installed at the University of Illinois, also has water blocks on the switch/hub networking components in the node.

The SuperMUC's 10PB disk array will run IBM's Global Parallel File System.

The German federal government and the state of Bavaria are kicking in €83m (£71.6m, $110.9m) to build SuperMUC at LRZ; that figure includes the cost of the hardware and software as well as operational costs, including the electricity bill, for five years. The Federal and state governments are also ponying up €50m (£43.1m, $66.8m) to build out the data centers at the LRZ facility in Garching to house the machinery.

The SuperMUC machine will be a serious upgrade for LRZ, and you have to believe that SGI was fighting hard to have the German HPC center upgrade its 62.3-teraflops Altix 4700, called HLRB-II, to a petaflops-class "UltraViolet" Altix UV super using Intel's Xeon 7500s and SGI's NUMAlink 5 interconnect. But for whatever reason, LRZ decided on the iDataPlex alternative using Xeon chips and a future InfiniBand switching technology called FDR, short for Fourteen Data Rate and offering 56GB/sec of bandwidth compared to the 40GB/sec Quad Data Rate (QDR) speed available today. ®