Feeds

US energy lab's pump-happy petaflopper goes green

Shiny new Xeon–Xeon Phi hybrid splashes into water-cooled data center

The Essential Guide to IT Transformation

The US Department of Energy's National Renewable Energy Laboratory has hooked up with HP, Intel, and partners to design a new hybrid supercomputer and an energy-efficient data center wrapper for it that will – among other green gains – use the exhaust heat from the supercomputer to heat adjacent offices.

After all, when you're the DoE, you can't be messing around running supercomputers or the data centers that house them in an inefficient manner. You have to lead by example.

The new machine, which has yet to be nicknamed, will be set up in a data center at the Energy Systems Integration Facility in Golden, Colorado, and will be significant for a number of reasons.

First, NREL will have its own behemoth cluster and won't rely on processing capacity at the "Red Sky" supercomputer, rated at 433.5 teraflops on the Linpack Fortran benchmark test, that's installed at Sandia National Laboratory down in New Mexico. That super was built by Sun Microsystems using quad-core Xeon 5500 processors back in 2009 when Sun was still in the HPC cluster business – Oracle is not interested in traditional, technical HPC.

The new machine is being built by HP in conjunction with Intel, which wants to use the box to showcase future Xeon processors and Xeon Phi x86 coprocessors, formerly known as "Many Integrated Core" or "Knights Corner" before they were officially branded Xeon Phi chips in June.

The plan, according to Marc Hamilton, vice president of high performance computing at HP, calls for NREL to spend $10m to get a petaflops-class machine that is "all Intel inside." The machine will be built in stages between now and the summer of 2013, when it will be completed and up to its full processing capacity.

In the first phase, NREL will take delivery of 1,600 server nodes from HP. These will be comprised of a mixture of ProLiant SL230s and SL250s Gen8 nodes, which slide into the SL6500 chassis and offer the densest computing that HP puts on the market.

The SL230s nodes (that "s" is part of the product name, not a plural) will be equipped with eight-core Xeon E5-2670 processors in the "Sandy Bridge" family that were announced in the spring, with 2GB per core or 32GB of main memory. The SL250 nodes will use a mix of the same processors, which spin at 2.6GHz, and Xeon Phi coprocessors, which plug into PCI-Express slots and are each expected to deliver at least 1 teraflops of double-precision floating point oomph.

The plan calls for an upgrade of the processors to future "Ivy Bridge" Xeon E5 chips and the addition of 600 Xeon Phi coprocessor cards to the cluster. Hamilton said the peak performance of the cluster, which will be lashed together with 56Gb/sec FDR InfiniBand switches and server adapters from Mellanox Technologies, will be "north of one petaflops of performance." The machine will run Linux and be attached to a Lustre-based clustered file system.

The SL230s is a half-width computing node that is 1U high; the SL250s is a half-width node that is 2U high. Both machines have only one PCI-Express 3.0 x8 slot, so the only reason to choose the SL250s is to give the Xeon Phi coprocessor a little more space in the chassis. You might think it's all about airflow, but Hamilton tells El Reg that one of the secret sauces in the new NREL petaflopper is component-level water cooling inside the server. Hamilton says that this cooling method has not yet been formally launched by HP, so he is not giving out names or details; he adds that it will be available commercially in the coming months.

"Pumps trump fans," says Steve Hammond, computational science center director at NREL. "A small pump moving water is more efficient at getting rid of heat than a slew of muffin fans."

Hammond tells El Reg that this water cooling was an integral part of the cluster and data-center design, which is expected to have a power usage effectiveness (PUE) ratio of 1.06. PUE is the ratio of the power consumed by the data center divided by the power consumed by the IT gear when it is running; a PUE of 1.06 and means the NREL design will have nearly all of its power being used by the gear, not for moving cold air around.

NREL looked at using mineral oil and other coolants to get rid of the heat in the cluster, but Hammond said he did not want to have to "get into a pair of coveralls to service a machine."

The DOE's NREL data center warms offices with data center heat

The DOE's NREL data center warms offices with data center heat (click to enlarge)

Hammond says that the data center will have no mechanical chillers and will only use evaporative cooling, which is a choice you can make in the dry Colorado air. The design that HP and Intel have come up for the SL server nodes is capable of taking away about 90 per cent of the heat generated by the servers, leaving that evaporative cooling to only have to cope with the remaining 10 per cent of the load on the server side.

The water-cooling system is configured to bring 75-degree water to the server components and to run with a 95-degree return temperature into the evaporative coolers that chill the water and the air inside the data center as well.

The NREL data center will have about 10,000 square feet of raised floor space, and the petaflops ceepie-phibie machine will only take up about 1,000 square feet of that – NREL has plenty of room to grow.

The data center and adjacent office design calls for waste heat pulled out of the data center to be used to warm offices in the colder months, and some excess heat may be used in other building throughout the Golden campus.

The new data center opens on October 18 and NREL takes delivery of the first server nodes on November 5.

The one thing that was not clear was where the renewable energy lab is getting its juice from – presumably not from burning dirty ol' coal. ®

The Essential Guide to IT Transformation

More from The Register

next story
Sysadmin Day 2014: Quick, there's still time to get the beers in
He walked over the broken glass, killed the thugs... and er... reconnected the cables*
Auntie remains MYSTIFIED by that weekend BBC iPlayer and website outage
Still doing 'forensics' on the caching layer – Beeb digi wonk
Microsoft says 'weird things' can happen during Windows Server 2003 migrations
Fix coming for bug that makes Kerberos croak when you run two domain controllers
Cisco says network virtualisation won't pay off everywhere
Another sign of strain in the Borg/VMware relationship?
VVOL update: Are any vendors NOT leaping into bed with VMware?
It's not yet been released but everyone thinks it's the dog's danglies
prev story

Whitepapers

Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Boost IT visibility and business value
How building a great service catalog relieves pressure points and demonstrates the value of IT service management.
Why and how to choose the right cloud vendor
The benefits of cloud-based storage in your processes. Eliminate onsite, disk-based backup and archiving in favor of cloud-based data protection.
The Essential Guide to IT Transformation
ServiceNow discusses three IT transformations that can help CIO's automate IT services to transform IT and the enterprise.
Maximize storage efficiency across the enterprise
The HP StoreOnce backup solution offers highly flexible, centrally managed, and highly efficient data protection for any enterprise.