Summit for the readers who are hot for petaFLOPs: Server nodes flashed at SC17

Oak Ridge Top 500-leading system's innards

By Chris Mellor

Posted in Servers, 23rd November 2017 18:09 GMT

Analysis IBM offered HPC fans at SC17 a gawk at the server tray for the upcoming Summit supercomputer at Oak Ridge National Laboratory (ORNL), Tennessee.

This is the system slated to knock China's 93 petaFLOPS Sunway TaihuLight system off the top of the supercomputer tree when it goes live. It is slated to pump out a hoped-for 200 petaFLOPS.

The Summit system follows on from ORNL's current 27 petaFLOP Titan system, computing 5-10 times faster, storing eight times more data and moving it 5-10 times faster as well. It will enable simulation models with finer resolution than Titan, meaning higher fidelity and more accurate simulations.

Summit will have around 4,600 server tray nodes, which will use IBM's Witherspoon Power S922LC trays.

SC17 Summit server tray tweet (

According to Tom's Hardware, these water-cooled trays feature a pair of POWER9 processors, each connected by a 150GB/sec NVLink 2.0 to three 7.5 teraFLOP NVIDIA Volta V100 accelerators (each with a GV100 GPU) which are inter-connected across the NVLink.

Volta GV100 GPU with 84 streaming multiprocessors

Both the CPUs and the GPUs are water-cooled. There is 300GB/sec of aggregated NVLink bandwidth.

The POWER9 CPUs have up to 24 cores and 96 threads. NVLink supports CPU mastering and cache coherence capabilities with IBM POWER9 CPU-based servers. The tray will have from 512GB to 2TB of coherent DDR4 memory, with 340GB/sec of memory bandwidth. All six GPUs and the two POWER9 CPUs can access main memory.

The system uses will use PCIe gen 4 and CAPI to hook up SSDs, FPGAs and NICS, and there is 1.TB of bust buffer NV-RAM.

Trays will be connected across Mellanox InfiniBand links, 100Gbit/s EDR.

Summit racks

The Summit machine will have up to 250PB of storage, accessed by Spectrum Scale (GPFS) and 2.5TB/sec of aggregate bandwidth. This is interfaced via the burst buffers.

Simplistically the data flow is from Spectrum Scales across InfiniBand and into a server node's memory. Each POWER9 CPU controls the activities of three GPUs and these eight compute entities access main memory and much data. The results are streamed out to the burst buffer and then pushed out to the GPFS storage.

Altogether the system will need 15MW of power and take up around 9,000 square feet of space. ORNL is installing it now. Get a Summit fact sheet here and FAQs here. ®

Sign up to our NewsletterGet IT in your inbox daily


More from The Register

IBM man goes deep on why they're all shiny OpenCAPI people

HPC Blog More details emerge at HPC powwow in Switzerland

SanDisk man tipped off his family to Fusion-io fusion, bagged $220k in share snatch – says SEC

US financial watchdog drags four into court

Google: We look forward to running non-Intel processors in our cloud

OK, so someone's angling for a discount

Why OpenCAPI is a declaration of interconnect fabric war

+Comment Any standard but Intel in another CPU-memory interconnect consortium

Ruskie boffins blasted for using nuke bomb lab's supercomputer to mine crypto-rubles

Kremlin goes nuclear on sly digi-cash-crafting eggheads

Blade Runner time: Retail replicant buys into WANdisco’s Fusion product

Firm starting to win big orders to its replication to the cloud product

WANdisco sticks Fusion into Amazon's Snowballs for mega-petabyte data pelt

Replication tech integrated with data truck - yes an actual truck...

Curse of Woz strikes again – first Fusion-io fizzles out, now Primary Data goes down

Startup goes silent, scent of burned cash wafts from offices

While USA is distracted by its President's antics, China is busy breaking another fusion record

Tweet a GIF about that, Donald

SciNet supercomputer's GPFS trick: We node what you did, burst buffer

Good news for Canadian HPC models