Original URL: https://www.theregister.com/2008/12/08/power7_bluewaters_data_center/
IBM drops Power7 drain in 'Blue Waters'
Super efficiency
Posted in Channel, 8th December 2008 20:57 GMT
A month ago, IBM and the University of Illinois broke ground for the data center that will eventually house the Power7-based "Blue Waters" massively parallel supercomputer. The data center, it turns out, is as tricky to design as the processor and server that will be humming along inside it.
During a talk at the recent SC2008 supercomputing event in Austin, Texas, someone said it would not be long before power companies would be giving away supercomputers to any governmental agency or corporation that signed a long-term power contract. This was a joke. But it brings into sharp focus the fact that power and cooling are the main limitations on scalability for supercomputers. That's been the case since the first Cray-1 vector super came out in 1976.
Stephen Elbert, a researcher at the Pacific Northwest National Laboratory, explained that there are plenty of data centers out there burning 60 to 70 megawatts and that a few have already broken through the 100 megawatts barrier. "Beyond that, you have to be your own power company." Or, to do a deal with one, as Google has done with the Bonneville Power Administration in its The Dalles, Oregon data center.
The biggest problem facing data centers, according to Ed Seminaro, chief Unix hardware architect at IBM and one of the people responsible for the Blue Waters project, is the inefficiency of the data center itself. Many of the tricks that IBM will be using in the data center that houses Blue Waters will probably be considered by other computing facilities and server makers. The reason is simple: There is no other way to get more computing done than focus on the data center.
By Seminaro's calculations, in the typical HPC data center, a 300 watt device running in a data center requires 800 watts of input power. "The efficiency is just awful," he said. From that 800 watts of input power required for a 300 watt server, the data center's power conversion hardware (including uninterruptible power supplies and AC distribution units) consumes 160 watts; cooling towers, condensers, evaporators, and air movers eat 140 watts. The cooling equipment inside the device (fans and blowers, but sometimes pumps for water blocks and jackets) eats another 50 watts.
The power supplies, which step down the juice so it can be consumed by the server itself, eats another 150 watts of that 800 watts of input power. To idealize this, it takes 2.67 units of server power to actually run a single server. And on a multi-petaflops performance scale, this kind of inefficiency gets very costly, very fast.
That's why the University of Illinois and the Blue Waters design is set to become the standard bearer for Power-based HPC server performance in the coming years, much as Lawrence Livermore National Lab has been for a decade with IBM's ASCI Purple and BlueGene/L supers.
IBM has been vague about the Blue Waters design, which seeks to create a multi-petaflops machine using approximately 200,000 Power7 processor cores. In the Blue Waters announcement back in July, all that IBM has said is that Power7 chips will be implemented in a 45 nanometer process and will have multiple cores, meaning more than two. But Ross Mauri, general manager of IBM's Power Systems division, has separately confirmed to me that Power7 will be an eight-core processor.
(He did not say what kind of cores would be in there. It could be a mix of Power and other vector cores, much like the Cell Power chip used in game consoles and in supercomputers today. This chip has a single 64-bit Power core and eight vector-style coprocessors).
According to an early presentation of the National Center for Supercomputing Applications (NCSA) at the University of Illinois, the Blue Eaters machine is supposed to have more than 800 TB of main memory (with at least 32 GB of main memory per SMP node). The initial goal is for 100 Gb/sec of external bandwidth, eventually quadrupling that to 400 Gb/sec.
Inside Blue Waters
The Blue Waters data center, according to Seminaro, will fit in around 10,000 square feet of space, comprised of 162 racks of servers, organized into three columns, with each column having six rows with nine racks each. The Power server that the Blue Waters machine is based on will be a kicker to the current Power 575, which has 16 dual-core Power6 processors crammed into a 2U chassis. This machine is notable in that it has water blocks right on the processors and main memory circuits and that it links right into the water cooling system of the data center. It doesn't rely on fans for cooling at all. (That 2U server is 24 inches wide, not the standard 19 inches, by the way).
IBM has not provided the specs of this future Power7 server, but it looks like IBM will put 64 cores in a 2U box, which is accomplished by putting two Power7 chips into a single ceramic package and sharing a single socket, with four sockets on the board. That's four times the cores and one half the number of sockets in the Power 575 available today. This 2U box is expected to be rated at around 2 teraflops and support up to 128 GB of main memory.
Those 64 Power7 cores per 2U box at around 2 teraflops provide considerably more oomph than the 602 gigaflops in the current Power 575 machine, which has only 16 cores. (And it looks like this expected rating for the Power7 chip is based on 4 GHz chip, if you do the math). Anyway, with 4 GHz cores, and 64 cores per box, that works out to Blue Waters being rated at 6.8 petaflops if IBM can cram 21 servers into a rack.
Back in July, people familiar with the Blue Waters project hinted to me that the clock speed on the Power7 chips might be more in the range of 3 GHz to 4 GHz, not the 4 GHz speed originally reported in El Reg, and they also said that these initial Power7 chips could actually be delivered in the Blue Waters machine sometime between the fall of 2009 and the spring of 2010, not all the way out in 2011. So with lower clock speeds, say 3 GHz, it might come in at only 5 petaflops or so.
As with the Power 575 "Hydro Cluster", as it is nicknamed, the Blue Waters machine will have chilled water from the building going right into the server. Instead of having external switching, the server will also have integrated InfiniBand switches for both server interconnect and for linking to storage. The Blue Waters data center will not have room-level air conditioning, either, and it will use water up to a 60 degree (F) temperature, which is sufficient to cool the racks of servers and saves on the energy bill back on the water chillers.
IBM will also be stepping down 13,000 volt AC power to 480 volt AC juice and feeding it directly into the servers. IBM could have distributed DC power directly to the servers, but NCSA didn't want to do that. It doesn't matter, through, because the Power7 server will have a single power supply that will span from 200 to 480 volts AC or from 400 to 600 volts DC, which means customers can change the way they distribute electricity and not have to change anything in the server.
The Blue Waters data center will also make use of air economizers, the HPC industry term for using outside air to help keep gear cool, something you can do in a temperate region - as Illinois certainly is - for many months of the year. The facility is designed to have about 17.6 megawatts of input power on the data center, supporting about 15 megawatts of IT load.
Seminaro says that IBM is estimating that the data center will have about 98 per cent power distribution efficiency and that depending on the time of the year, the cooling infrastructure will run at between 80 and 93 per cent efficiency. (In cooler weather, it will be more efficient thanks to the air economizers).
IBM expects the overall efficiency of the data center housing Blue Waters to be around 85 percent. So to power up a 300-watt device in this data center will only take about 350 watts of input power. That's a very big improvement. ®