Feeds

US energy lab's pump-happy petaflopper goes green

Shiny new Xeon–Xeon Phi hybrid splashes into water-cooled data center

7 Elements of Radically Simple OS Migration

The US Department of Energy's National Renewable Energy Laboratory has hooked up with HP, Intel, and partners to design a new hybrid supercomputer and an energy-efficient data center wrapper for it that will – among other green gains – use the exhaust heat from the supercomputer to heat adjacent offices.

After all, when you're the DoE, you can't be messing around running supercomputers or the data centers that house them in an inefficient manner. You have to lead by example.

The new machine, which has yet to be nicknamed, will be set up in a data center at the Energy Systems Integration Facility in Golden, Colorado, and will be significant for a number of reasons.

First, NREL will have its own behemoth cluster and won't rely on processing capacity at the "Red Sky" supercomputer, rated at 433.5 teraflops on the Linpack Fortran benchmark test, that's installed at Sandia National Laboratory down in New Mexico. That super was built by Sun Microsystems using quad-core Xeon 5500 processors back in 2009 when Sun was still in the HPC cluster business – Oracle is not interested in traditional, technical HPC.

The new machine is being built by HP in conjunction with Intel, which wants to use the box to showcase future Xeon processors and Xeon Phi x86 coprocessors, formerly known as "Many Integrated Core" or "Knights Corner" before they were officially branded Xeon Phi chips in June.

The plan, according to Marc Hamilton, vice president of high performance computing at HP, calls for NREL to spend $10m to get a petaflops-class machine that is "all Intel inside." The machine will be built in stages between now and the summer of 2013, when it will be completed and up to its full processing capacity.

In the first phase, NREL will take delivery of 1,600 server nodes from HP. These will be comprised of a mixture of ProLiant SL230s and SL250s Gen8 nodes, which slide into the SL6500 chassis and offer the densest computing that HP puts on the market.

The SL230s nodes (that "s" is part of the product name, not a plural) will be equipped with eight-core Xeon E5-2670 processors in the "Sandy Bridge" family that were announced in the spring, with 2GB per core or 32GB of main memory. The SL250 nodes will use a mix of the same processors, which spin at 2.6GHz, and Xeon Phi coprocessors, which plug into PCI-Express slots and are each expected to deliver at least 1 teraflops of double-precision floating point oomph.

The plan calls for an upgrade of the processors to future "Ivy Bridge" Xeon E5 chips and the addition of 600 Xeon Phi coprocessor cards to the cluster. Hamilton said the peak performance of the cluster, which will be lashed together with 56Gb/sec FDR InfiniBand switches and server adapters from Mellanox Technologies, will be "north of one petaflops of performance." The machine will run Linux and be attached to a Lustre-based clustered file system.

The SL230s is a half-width computing node that is 1U high; the SL250s is a half-width node that is 2U high. Both machines have only one PCI-Express 3.0 x8 slot, so the only reason to choose the SL250s is to give the Xeon Phi coprocessor a little more space in the chassis. You might think it's all about airflow, but Hamilton tells El Reg that one of the secret sauces in the new NREL petaflopper is component-level water cooling inside the server. Hamilton says that this cooling method has not yet been formally launched by HP, so he is not giving out names or details; he adds that it will be available commercially in the coming months.

"Pumps trump fans," says Steve Hammond, computational science center director at NREL. "A small pump moving water is more efficient at getting rid of heat than a slew of muffin fans."

Hammond tells El Reg that this water cooling was an integral part of the cluster and data-center design, which is expected to have a power usage effectiveness (PUE) ratio of 1.06. PUE is the ratio of the power consumed by the data center divided by the power consumed by the IT gear when it is running; a PUE of 1.06 and means the NREL design will have nearly all of its power being used by the gear, not for moving cold air around.

NREL looked at using mineral oil and other coolants to get rid of the heat in the cluster, but Hammond said he did not want to have to "get into a pair of coveralls to service a machine."

The DOE's NREL data center warms offices with data center heat

The DOE's NREL data center warms offices with data center heat (click to enlarge)

Hammond says that the data center will have no mechanical chillers and will only use evaporative cooling, which is a choice you can make in the dry Colorado air. The design that HP and Intel have come up for the SL server nodes is capable of taking away about 90 per cent of the heat generated by the servers, leaving that evaporative cooling to only have to cope with the remaining 10 per cent of the load on the server side.

The water-cooling system is configured to bring 75-degree water to the server components and to run with a 95-degree return temperature into the evaporative coolers that chill the water and the air inside the data center as well.

The NREL data center will have about 10,000 square feet of raised floor space, and the petaflops ceepie-phibie machine will only take up about 1,000 square feet of that – NREL has plenty of room to grow.

The data center and adjacent office design calls for waste heat pulled out of the data center to be used to warm offices in the colder months, and some excess heat may be used in other building throughout the Golden campus.

The new data center opens on October 18 and NREL takes delivery of the first server nodes on November 5.

The one thing that was not clear was where the renewable energy lab is getting its juice from – presumably not from burning dirty ol' coal. ®

Best practices for enterprise data

More from The Register

next story
Sysadmin Day 2014: Quick, there's still time to get the beers in
He walked over the broken glass, killed the thugs... and er... reconnected the cables*
VMware builds product executables on 50 Mac Minis
And goes to the Genius Bar for support
Multipath TCP speeds up the internet so much that security breaks
Black Hat research says proposed protocol will bork network probes, flummox firewalls
Auntie remains MYSTIFIED by that weekend BBC iPlayer and website outage
Still doing 'forensics' on the caching layer – Beeb digi wonk
Microsoft's Euro cloud darkens: US FEDS can dig into foreign servers
They're not emails, they're business records, says court
Microsoft says 'weird things' can happen during Windows Server 2003 migrations
Fix coming for bug that makes Kerberos croak when you run two domain controllers
Cisco says network virtualisation won't pay off everywhere
Another sign of strain in the Borg/VMware relationship?
prev story

Whitepapers

7 Elements of Radically Simple OS Migration
Avoid the typical headaches of OS migration during your next project by learning about 7 elements of radically simple OS migration.
Implementing global e-invoicing with guaranteed legal certainty
Explaining the role local tax compliance plays in successful supply chain management and e-business and how leading global brands are addressing this.
Consolidation: The Foundation for IT Business Transformation
In this whitepaper learn how effective consolidation of IT and business resources can enable multiple, meaningful business benefits.
Solving today's distributed Big Data backup challenges
Enable IT efficiency and allow a firm to access and reuse corporate information for competitive advantage, ultimately changing business outcomes.
A new approach to endpoint data protection
What is the best way to ensure comprehensive visibility, management, and control of information on both company-owned and employee-owned devices?