Feeds

Oak Ridge changes Jaguar's spots from CPUs to GPUs

Nuke lab's super upgraded to a 20-petaflops Titan

Secure remote control for conventional and virtual desktops

The mystery surrounding the architecture of the future "Titan" supercomputer to be installed at Oak Ridge National Laboratories and funded by the US Department of Energy is over.

Oak Ridge, which is the US nuke lab most associated with open (rather than classified and often military) science, has been working with Cray and Nvidia on future hybrid CPU-GPU designs for several years and is Cray's keystone HPC customer. So it is no surprise that the Titan supercomputer will marry Cray's Opteron-based supers with Nvidia's GPU coprocessors to make a more energy-efficient super – weighing in at a top speed of 20 petaflops when (and if) it is fully extended with GPUs.

Oak Ridge Titan logo

Rather than just throwing out the existing "Jaguar" supercomputer, Oak Ridge is doing what it has done in the past – and what the Cray supercomputers are designed to undergo – a timely upgrade. Oak Ridge did a two-step upgrade from XT3 to Xt4 systems for around $200m to get to the initial Jaguar configuration, which was rated at 263 teraflops of sustained performance on the Linpack Fortran benchmark test commonly used to rank the performance.

The current Jaguar machine consists of a mix of Cray XT4 and XT5 cabinets using six-core Opteron HE processors with a total of 224,256 cores and 362TB of main memory. The Jaguar server nodes are linked by the "SeaStar2+" XT interconnect and deliver 2.33 peak theoretical petaflops and 1.76 petaflops sustained on the Linpack test. This machine has 18,688 nodes, each with two Opteron processors.

The Titan system, also known as the OLCF-3 system in Oak Ridge-ese, will be based on the Cray XK6 ceepie-geepie, announced in May. The XK6 swaps out one of the Opteron processor sockets on the four-node XE system blade and puts in four of Nvidia's Tesla X2090 GPU coprocessors and a chipset that implements a PCI-Express 2.0 link between the Opteron G34 sockets and the Nvidia GPUs.

Each Opteron socket has four memory slots sporting DDR3 main memory. (Cray currently supports 2GB and 4GB sticks, but can use fatter memory if customers wish to do so.) Cray is not shipping the XK6 machines with the existing 12-core "Magny-Cours" Opteron 6100 processors, but is rather waiting for Advanced Micro Devices to get the 16-core "Interlagos" Opteron 6200 chips out the door. AMD has begun shipping them to OEM customers (and presumably Cray is at the front of the line), but the Opteron 6200s have not been formally announced yet.

Cray XK6 supercomputer blade server

Cray's Tesla X2090-equipped XK6 system board

The XK6 is based on the "Gemini" XE interconnect, just like the XE6 Opteron-only supers upon which they are based. That interconnect, which debuted in May 2010, is the heart of the XE6 and XE6m supers, implemented in 3D torus and 2D torus interconnects respectively on those two families of machines.

The Gemini interconnect has a lot more bandwidth than the SeaStar2+ interconnect and with about one-third the latency for hops between adjacent nodes, which results in the XE6 machine being able to pass about 100 times the messages between nodes. This is key for the message passing interface (MPI) protocol underlying most parallel supercomputer apps. And as you can now see, it will be key for the XK6 ceepie-geepie machines, since the GPUs are going to be able to do more math than the CPUs they are lashed to. A lot more messages are going to be flying around as calculations get done more quickly.

Under the plan Oak Ridge has put together to upgrade the Jaguar system to the Titan machine, the first thing that will be done is that all of those 18,688 two-socket nodes will be replaced with 4,672 four-socket hybrid XK6 blade servers. The net result will be a machine with 299,008 Opteron cores and – depending on the clock speeds AMD can deliver – probably somewhere between 25 and 30 per cent more raw x86 flops in the system ... and presumably better performance given the faster interconnect. The machine will also have 600TB of main memory, which will not hurt performance, either.

Before the end of this year, 960 of the current Tesla X2090 GPUs will be added to the Titan system (in about one-twentieth of the system nodes). Eventually the future GPU coprocessors from Nvidia based on the "Kepler" GPUs – due later this year if all goes well – will also be added to the machine, according to Steve Scott, who is the new CTO of the Tesla GPU line at Nvidia.

Intelligent flash storage arrays

More from The Register

next story
Scrapping the Human Rights Act: What about privacy and freedom of expression?
Justice minister's attack to destroy ability to challenge state
WHY did Sunday Mirror stoop to slurping selfies for smut sting?
Tabloid splashes, MP resigns - but there's a BIG copyright issue here
Google hits back at 'Dear Rupert' over search dominance claims
Choc Factory sniffs: 'We're not pirate-lovers - also, you publish The Sun'
EU to accuse Ireland of giving Apple an overly peachy tax deal – report
Probe expected to say single-digit rate was unlawful
Inequality increasing? BOLLOCKS! You heard me: 'Screw the 1%'
There's morality and then there's economics ...
Hey Brit taxpayers. You just spent £4m on Central London ‘innovation playground’
Catapult me a Mojito, I feel an Digital Innovation coming on
While you queued for an iPhone 6, Apple's Cook sold shares worth $35m
Right before the stock took a 3.8% dive amid bent and broken mobe drama
EU probes Google’s Android omerta again: Talk now, or else
Spill those Android secrets, or we’ll fine you
prev story

Whitepapers

Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Storage capacity and performance optimization at Mizuno USA
Mizuno USA turn to Tegile storage technology to solve both their SAN and backup issues.
The next step in data security
With recent increased privacy concerns and computers becoming more powerful, the chance of hackers being able to crack smaller-sized RSA keys increases.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.
A strategic approach to identity relationship management
ForgeRock commissioned Forrester to evaluate companies’ IAM practices and requirements when it comes to customer-facing scenarios versus employee-facing ones.