A 16 Petaflop Cray: The key to fantastic summer barbecues

The big switch

The Met Office’s next big milestone in its super project comes this summer, when workloads on the IBM Power machine are finally run on to the Cray and no longer in parallel. The plan is to run workloads in parallel for a few months to ensure there’s no problems and that performance and calculations land as expected.

This first XC40 is due to be operational by the end of September; the second machine by March 2016.

Thirty applications are moving: the biggest is two million lines, others at 1.5 million, and smaller apps are 20-30,000 lines – all in Fortran.

Cray is working with the Met’s engineers on the move: scientific software engineers are arranged in groups around applications. Some in the teams are more on the IT side, while others tend towards the science end of the spectrum.

The trickiest part is performance – trying to spot bottlenecks, understanding what’s caused them and how to fix or work around them.

Underwood reckons migration should be relatively simple – at any rate, more so than during the Met’s last big platform shift, from an NEC FX 8 Vector to that IBM Power system five years ago: that meant going parallel and moving from different processor architectures.

By far the greatest challenge has been in trying to build that third and final system, which is due in 2017. The final word on exact performance numbers, the architecture – even the finer details of the brand-new computer room to house this beast — are pending the release of Intel's Skylake.

Underwood could go Haswell or Broadwell, used in the first XC40s, but his heart is set on that duel- and quad-core Skylake family of CPUs – expected at the Intel’s Developer Forum later this year.

On current spec, the XC40 can stack up to 384 64-bit Xeon E5 processors per cabinet, with between 64GB and 256GB per node.

Performance peaks at up to 226 Teraflops per system cabinet, and connection is via Infiniband on I/O and 40 and 10GB Ethernet, Fibre Channel and Ethernet. Each cabinet weighs in at 3,450lbs.

The first two systems will feature two identical machines with four cabinets, 544 compute nodes per machines with 35,000 cores in total. The second adds five times more compute power. Connection will be via Cray’s Aries high-speed network, four Sonexion and 2000 file systems delivering 12Pb.

At time of writing, contracts were in the final stages of being signed off on the new computer hall, with ground in Exeter about to be broken.

But with Intel's latest chipset not in the real world, nobody seems willing to commit on performance of the Met's software models or what the new computer hall will need from either a power or a cooling perspective.

Underwood said: “Cray has a close relationship with Intel with regard to the chips... So we very much rely on their closeness of understanding. We meet regularly with Intel but as a very large consumer of their processors, which Cray is, it gets the latest update."

Underwood promised the new hall would, like the first, feature free-air cooling, cold-air containment and there'd be water cooling. Power will be DC, to cut costs of converting from AC, and it will be supplemented by solar to hit a target PuE of 1.2 to 1.3 – similar to current levels.

“We have lots of ideas, but until we know the horsepower of the engine that is the chip and the quantities, that’s when we do the final design,” Underwood said.

The goal is to operate the halls as efficiently and cost effectively as possible, in keeping with the first IT halls.

Expectations are huge for the Cray trio. The Met Office has been forecasting weather since 1854. Arguably now – with possibly the exception of the part it played in wartime military operations such as the Allied landings in Normandy – the Met Office’s role is more important than ever.

The human, financial and social impact of climate change and severe weather on profits and losses, food production, safety and politics are putting ever more pressure on forecasters to deliver detailed insight.

Will weather forecasting improve and get more reliable after 2017? The UK's weather is famously unpredictable thanks to our location – a small island to the west of a large continent and with a large ocean on the other side, which combine to produce a very dynamic weather system.

“The key thing is to be able to accurately predict change,” Underwood tells us. “We are in the forecasting business. If we could tell you 100 per cent what was happening, we wouldn’t call it forecasting. What we can do is become more certain of a robustness of a range of forecasts." ®