Feeds

Oak Ridge changes Jaguar's spots from CPUs to GPUs

Nuke lab's super upgraded to a 20-petaflops Titan

3 Big data security analytics techniques

The mystery surrounding the architecture of the future "Titan" supercomputer to be installed at Oak Ridge National Laboratories and funded by the US Department of Energy is over.

Oak Ridge, which is the US nuke lab most associated with open (rather than classified and often military) science, has been working with Cray and Nvidia on future hybrid CPU-GPU designs for several years and is Cray's keystone HPC customer. So it is no surprise that the Titan supercomputer will marry Cray's Opteron-based supers with Nvidia's GPU coprocessors to make a more energy-efficient super – weighing in at a top speed of 20 petaflops when (and if) it is fully extended with GPUs.

Oak Ridge Titan logo

Rather than just throwing out the existing "Jaguar" supercomputer, Oak Ridge is doing what it has done in the past – and what the Cray supercomputers are designed to undergo – a timely upgrade. Oak Ridge did a two-step upgrade from XT3 to Xt4 systems for around $200m to get to the initial Jaguar configuration, which was rated at 263 teraflops of sustained performance on the Linpack Fortran benchmark test commonly used to rank the performance.

The current Jaguar machine consists of a mix of Cray XT4 and XT5 cabinets using six-core Opteron HE processors with a total of 224,256 cores and 362TB of main memory. The Jaguar server nodes are linked by the "SeaStar2+" XT interconnect and deliver 2.33 peak theoretical petaflops and 1.76 petaflops sustained on the Linpack test. This machine has 18,688 nodes, each with two Opteron processors.

The Titan system, also known as the OLCF-3 system in Oak Ridge-ese, will be based on the Cray XK6 ceepie-geepie, announced in May. The XK6 swaps out one of the Opteron processor sockets on the four-node XE system blade and puts in four of Nvidia's Tesla X2090 GPU coprocessors and a chipset that implements a PCI-Express 2.0 link between the Opteron G34 sockets and the Nvidia GPUs.

Each Opteron socket has four memory slots sporting DDR3 main memory. (Cray currently supports 2GB and 4GB sticks, but can use fatter memory if customers wish to do so.) Cray is not shipping the XK6 machines with the existing 12-core "Magny-Cours" Opteron 6100 processors, but is rather waiting for Advanced Micro Devices to get the 16-core "Interlagos" Opteron 6200 chips out the door. AMD has begun shipping them to OEM customers (and presumably Cray is at the front of the line), but the Opteron 6200s have not been formally announced yet.

Cray XK6 supercomputer blade server

Cray's Tesla X2090-equipped XK6 system board

The XK6 is based on the "Gemini" XE interconnect, just like the XE6 Opteron-only supers upon which they are based. That interconnect, which debuted in May 2010, is the heart of the XE6 and XE6m supers, implemented in 3D torus and 2D torus interconnects respectively on those two families of machines.

The Gemini interconnect has a lot more bandwidth than the SeaStar2+ interconnect and with about one-third the latency for hops between adjacent nodes, which results in the XE6 machine being able to pass about 100 times the messages between nodes. This is key for the message passing interface (MPI) protocol underlying most parallel supercomputer apps. And as you can now see, it will be key for the XK6 ceepie-geepie machines, since the GPUs are going to be able to do more math than the CPUs they are lashed to. A lot more messages are going to be flying around as calculations get done more quickly.

Under the plan Oak Ridge has put together to upgrade the Jaguar system to the Titan machine, the first thing that will be done is that all of those 18,688 two-socket nodes will be replaced with 4,672 four-socket hybrid XK6 blade servers. The net result will be a machine with 299,008 Opteron cores and – depending on the clock speeds AMD can deliver – probably somewhere between 25 and 30 per cent more raw x86 flops in the system ... and presumably better performance given the faster interconnect. The machine will also have 600TB of main memory, which will not hurt performance, either.

Before the end of this year, 960 of the current Tesla X2090 GPUs will be added to the Titan system (in about one-twentieth of the system nodes). Eventually the future GPU coprocessors from Nvidia based on the "Kepler" GPUs – due later this year if all goes well – will also be added to the machine, according to Steve Scott, who is the new CTO of the Tesla GPU line at Nvidia.

Combat fraud and increase customer satisfaction

More from The Register

next story
MtGox chief Karpelès refuses to come to US for g-men's grilling
Bitcoin baron says he needs another lawyer for FinCEN chat
Did a date calculation bug just cost hard-up Co-op Bank £110m?
And just when Brit banking org needs £400m to stay afloat
One year on: diplomatic fail as Chinese APT gangs get back to work
Mandiant says past 12 months shows Beijing won't call off its hackers
Don't let no-hire pact suit witnesses call Steve Jobs a bullyboy, plead Apple and Google
'Irrelevant' character evidence should be excluded – lawyers
EFF: Feds plan to put 52 MILLION FACES into recognition database
System would identify faces as part of biometrics collection
Lavabit loses contempt of court appeal over protecting Snowden, customers
Judges rule complaints about government power are too little, too late
Ex-Tony Blair adviser is new top boss at UK spy-hive GCHQ
Robert Hannigan to replace Sir Iain Lobban in the autumn
Alphadex fires back at British Gas with overcharging allegation
Brit colo outfit says it paid for 347KVA, has been charged for 1940KVA
Jack the RIPA: Blighty cops ignore law, retain innocents' comms data
Prime minister: Nothing to see here, go about your business
Banks slap Olympus with £160 MEEELLION lawsuit
Scandal hit camera maker just can't shake off its past
prev story

Whitepapers

Designing a defence for mobile apps
In this whitepaper learn the various considerations for defending mobile applications; from the mobile application architecture itself to the myriad testing technologies needed to properly assess mobile applications risk.
3 Big data security analytics techniques
Applying these Big Data security analytics techniques can help you make your business safer by detecting attacks early, before significant damage is done.
Five 3D headsets to be won!
We were so impressed by the Durovis Dive headset we’ve asked the company to give some away to Reg readers.
The benefits of software based PBX
Why you should break free from your proprietary PBX and how to leverage your existing server hardware.
Securing web applications made simple and scalable
In this whitepaper learn how automated security testing can provide a simple and scalable way to protect your web applications.