Swiss boffins jump in Lake Lugano for Cray super
Pumping water uphill to cool a 750 teraflops Cascade
The boffins at the Swiss National Supercomputing Centre (CSCS) are opened up a new supercomputing center in Lugano this year, and now they are getting a shiny new "Cascade" parallel supercomputer from Cray to hum away inside of it.
The Cascade machine at CSCS is not just interesting because of the "Aries" interconnect that will glue its x86 server nodes together so they can share work. It is also notable because it will be cooled by Lake Lugano, the glacial lake that sits on the border between Switzerland and Italy.
Cray has not announced the Cascade machines yet, but here's what we do know. The new Aries interconnect links nodes together through a much-improved high-radix router that speaks to server nodes through a PCI-Express bus.
Given that the initial Cascade machines are based on Xeon E5 processors from Intel, and that these processors have PCI-Express 3.0 controllers right on the chip, everyone expects that only processors that support PCI-Express 3.0 links will plug into the Cascade boxes. So that means for now, the Opteron 6200s from Advanced Micro Devices are outta luck.
The Aries interconnect and Cascade system design were funded by the US Defense Advanced Research Projects Agency, so it has dibs on the first machine manufactured by Cray, which is expected to be delivered later this year. The Cascade line will be commercialized in 2013.
It looks like CSCS is next in line after that, with Cray saying that its Cascade machine will be installed later this year. That machine does not yet have a nickname, but if history is any guide, it will have an Alps mountain as its namesake.
This Cascade box will have in excess of 750 teraflops of peak performance, according to Cray, which tells El Reg that this particular initial configuration will be entirely based on Xeon processors. With the top-bin Xeon E5 part weighing in at 172 gigaflops per socket, and a server node having two sockets, this Cascade machine should have around 2,180 nodes.
The CSCS machine will not have either Intel Xeon Phi or Nvidia Tesla K20 coprocessors. These two coprocessors are, of course, options on the Cascade box, and it is reasonable to assume that at some point AMD Opterons and maybe even FirePro discrete graphics cards might also be options.
The CSCS supercomputer center in Lugano
The CSCS super will also be equipped with Cray's Sonexion storage arrays, which run the Lustre cluster file system. The word on the street is that the machine will have north of 1PB of capacity.
The Swiss boffins say this is only the first step in building up their HPC compute capacity, which will drive research in materials science, climate modeling, geosciences, molecular biology, and biomedical engineering.
You can watch a time-lapsed video of the new data center being constructed here , which took from February 2010 through January 2011 to complete. The site was an old bus depot, and it is a concrete cube that has reconfigurable floors designed to be able to support supercomputers for the next 40 years.
At 45 meters below the surface of Lake Lugano, the water is only 6 degrees Celsius, and it is this natural source of coolness that the future Cascade super will be chilled with. The data center is 2.8 kilometers from the lake, and has to be pushed up an elevation of 30 meters from the lake level.
A little more than half of the water pumped uphill goes to the data center, while the remainder goes to fill a reservoir; the pumps can push 760 liters of water per second.
The pumps to move lake water to the CSCS data center in Lugano
The cool lake water is brought into the data center, where it flows through heat exchangers, picking up the warmth from the running system. There are two pipes coming into the data center: one that can handle about 14 megawatts of cooling and the second adds another 7 megawatts.
When CSCS says it is trying to future proof, it means it. The water taken out of the data center is put into a cooling pond and is not put back into the lake until it cools down to at least 25 degrees Celsius.
Birds are going to love that pond; fish, not so much.
CSCS has been moving its computers into the new center since April, and a lot of that existing gear has Cray labels on it. The Swiss boffins were first in line to buy an XMT-2 massively threaded super back in February 2011  – called "Matterhorn," of course. The XMT-2 was commercialized by Cray as the Urika big data appliance  this year.
The 20-rack "Monte Rosa" Cray XE6 system was first installed as an XT5 back in June 2009 , but has had its performance more than doubled with an upgrade to an XE6 machine that is rated at 402 teraflops.
CSCS also has a two-rack XK6 hybrid CPU-GPU box, called "Tödi," that has 176 Opteron nodes with one Tesla X2090 GPU coprocessor on each node, and an IBM iDataPlex ceepie-geepie called "Castor" that has a modest sixteen Xeon 5600 nodes with two M2090 GPU coprocessors per node.
The Swiss boffins also have two smaller XE6 clusters and one aged Sun Fire cluster from back when Sun Microsystems was an independent company and cared about supercomputing.
Bootnote: Dominik Ulmer, general manager of CSCS, notified El Reg after this story ran that the 750 teraflopper will be named after "Piz Daint", after a mountain in the Val Müstair area close to the Swiss National Park.®