Feeds

US energy lab's pump-happy petaflopper goes green

Shiny new Xeon–Xeon Phi hybrid splashes into water-cooled data center

Intelligent flash storage arrays

The US Department of Energy's National Renewable Energy Laboratory has hooked up with HP, Intel, and partners to design a new hybrid supercomputer and an energy-efficient data center wrapper for it that will – among other green gains – use the exhaust heat from the supercomputer to heat adjacent offices.

After all, when you're the DoE, you can't be messing around running supercomputers or the data centers that house them in an inefficient manner. You have to lead by example.

The new machine, which has yet to be nicknamed, will be set up in a data center at the Energy Systems Integration Facility in Golden, Colorado, and will be significant for a number of reasons.

First, NREL will have its own behemoth cluster and won't rely on processing capacity at the "Red Sky" supercomputer, rated at 433.5 teraflops on the Linpack Fortran benchmark test, that's installed at Sandia National Laboratory down in New Mexico. That super was built by Sun Microsystems using quad-core Xeon 5500 processors back in 2009 when Sun was still in the HPC cluster business – Oracle is not interested in traditional, technical HPC.

The new machine is being built by HP in conjunction with Intel, which wants to use the box to showcase future Xeon processors and Xeon Phi x86 coprocessors, formerly known as "Many Integrated Core" or "Knights Corner" before they were officially branded Xeon Phi chips in June.

The plan, according to Marc Hamilton, vice president of high performance computing at HP, calls for NREL to spend $10m to get a petaflops-class machine that is "all Intel inside." The machine will be built in stages between now and the summer of 2013, when it will be completed and up to its full processing capacity.

In the first phase, NREL will take delivery of 1,600 server nodes from HP. These will be comprised of a mixture of ProLiant SL230s and SL250s Gen8 nodes, which slide into the SL6500 chassis and offer the densest computing that HP puts on the market.

The SL230s nodes (that "s" is part of the product name, not a plural) will be equipped with eight-core Xeon E5-2670 processors in the "Sandy Bridge" family that were announced in the spring, with 2GB per core or 32GB of main memory. The SL250 nodes will use a mix of the same processors, which spin at 2.6GHz, and Xeon Phi coprocessors, which plug into PCI-Express slots and are each expected to deliver at least 1 teraflops of double-precision floating point oomph.

The plan calls for an upgrade of the processors to future "Ivy Bridge" Xeon E5 chips and the addition of 600 Xeon Phi coprocessor cards to the cluster. Hamilton said the peak performance of the cluster, which will be lashed together with 56Gb/sec FDR InfiniBand switches and server adapters from Mellanox Technologies, will be "north of one petaflops of performance." The machine will run Linux and be attached to a Lustre-based clustered file system.

The SL230s is a half-width computing node that is 1U high; the SL250s is a half-width node that is 2U high. Both machines have only one PCI-Express 3.0 x8 slot, so the only reason to choose the SL250s is to give the Xeon Phi coprocessor a little more space in the chassis. You might think it's all about airflow, but Hamilton tells El Reg that one of the secret sauces in the new NREL petaflopper is component-level water cooling inside the server. Hamilton says that this cooling method has not yet been formally launched by HP, so he is not giving out names or details; he adds that it will be available commercially in the coming months.

"Pumps trump fans," says Steve Hammond, computational science center director at NREL. "A small pump moving water is more efficient at getting rid of heat than a slew of muffin fans."

Hammond tells El Reg that this water cooling was an integral part of the cluster and data-center design, which is expected to have a power usage effectiveness (PUE) ratio of 1.06. PUE is the ratio of the power consumed by the data center divided by the power consumed by the IT gear when it is running; a PUE of 1.06 and means the NREL design will have nearly all of its power being used by the gear, not for moving cold air around.

NREL looked at using mineral oil and other coolants to get rid of the heat in the cluster, but Hammond said he did not want to have to "get into a pair of coveralls to service a machine."

The DOE's NREL data center warms offices with data center heat

The DOE's NREL data center warms offices with data center heat (click to enlarge)

Hammond says that the data center will have no mechanical chillers and will only use evaporative cooling, which is a choice you can make in the dry Colorado air. The design that HP and Intel have come up for the SL server nodes is capable of taking away about 90 per cent of the heat generated by the servers, leaving that evaporative cooling to only have to cope with the remaining 10 per cent of the load on the server side.

The water-cooling system is configured to bring 75-degree water to the server components and to run with a 95-degree return temperature into the evaporative coolers that chill the water and the air inside the data center as well.

The NREL data center will have about 10,000 square feet of raised floor space, and the petaflops ceepie-phibie machine will only take up about 1,000 square feet of that – NREL has plenty of room to grow.

The data center and adjacent office design calls for waste heat pulled out of the data center to be used to warm offices in the colder months, and some excess heat may be used in other building throughout the Golden campus.

The new data center opens on October 18 and NREL takes delivery of the first server nodes on November 5.

The one thing that was not clear was where the renewable energy lab is getting its juice from – presumably not from burning dirty ol' coal. ®

Choosing a cloud hosting partner with confidence

More from The Register

next story
'Kim Kardashian snaps naked selfies with a BLACKBERRY'. *Twitterati gasps*
More alleged private, nude celeb pics appear online
Wanna keep your data for 1,000 YEARS? No? Hard luck, HDS wants you to anyway
Combine Blu-ray and M-DISC and you get this monster
US boffins demo 'twisted radio' mux
OAM takes wireless signals to 32 Gbps
Google+ GOING, GOING ... ? Newbie Gmailers no longer forced into mandatory ID slurp
Mountain View distances itself from lame 'network thingy'
EMC, HP blockbuster 'merger' shocker comes a cropper
Stand down, FTC... you can put your feet up for a bit
Apple flops out 2FA for iCloud in bid to stop future nude selfie leaks
Millions of 4chan users howl with laughter as Cupertino slams stable door
Students playing with impressive racks? Yes, it's cluster comp time
The most comprehensive coverage the world has ever seen. Ever
Run little spreadsheet, run! IBM's Watson is coming to gobble you up
Big Blue's big super's big appetite for big data in big clouds for big analytics
prev story

Whitepapers

Providing a secure and efficient Helpdesk
A single remote control platform for user support is be key to providing an efficient helpdesk. Retain full control over the way in which screen and keystroke data is transmitted.
A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.
Saudi Petroleum chooses Tegile storage solution
A storage solution that addresses company growth and performance for business-critical applications of caseware archive and search along with other key operational systems.
WIN a very cool portable ZX Spectrum
Win a one-off portable Spectrum built by legendary hardware hacker Ben Heck
The next step in data security
With recent increased privacy concerns and computers becoming more powerful, the chance of hackers being able to crack smaller-sized RSA keys increases.