Data Centre


Brit uni builds its own supercomputer from secondhand parts

She may not look like much, but she's got it where it counts, kid

By Chris Mellor


Durham University has built itself a secondhand supercomputer from recycled parts and beefed up its contribution to DiRAC (distributed research utilising advanced computing), the integrated facility for theoretical modelling and HPC-based research in particle physics, astronomy and cosmology.

The Institute for Cosmological Computing (ICC) at Durham, in northeast England, runs a COSMA5 system as its DiRAC contribution.

There are five DiRAC installations in the UK, which is a world leader in these HPC fields:

The Durham cluster listed above is a COSMA5 system, which features 420 IBM iDataPlex dx360 M4 servers with a 6m720 2.6 GHz Intel Sandy Bridge Xeon E5-2670 CPU cores. There is 53.76TB of DDR3 RAM and Mellanox FDR10 Infiniband in a 2:1 blocking configuration.

It has 2.5PB of DDN storage with two SD12K controllers configured in fully redundant mode. It's served by six GPFS servers connected into the controllers over full FDR and using RDMA over the FDR10 network into the compute cluster. COSMA5 uses the GPFS file system with LSF as its job scheduler.

The ICC and DiRAC needed to strengthen this system and found that the Hartree Centre at Daresbury had a supercomputer it needed rid of. This HPC system was installed in April 2012 but had to go because Daresbury had newer kit.

Durham had a machine room with power and cooling that could take it. Even better, its configuration was remarkably similar to COSMA5.

So HPC, storage and data analytics integrator OCF, and server relocation and data centre migration specialist Technimove dismantled, transported, and rebuilt the machine at the ICC. The whole exercise was funded by the Science and Technology Facilities Council.

COSMA6 arrived at Durham in April 2016, and was installed and tested at the ICC. It now extends Durham's DiRAC system as part of DiRAC 2.5.

COSMA6 has:

The Lustre filesystem and SLURM are used for its job submission system.

COSMA6 racks

Lydia Heck, ICC technical director, said: "While it was quite an effort to bring it to its current state, as it is the same architecture and the same network layout as our previous system, we expect this to run very well."

Durham now has both COSMA5 (6,500 cores) and COSMA6 (8,000 cores) contributing to DiRAC and available for researchers.

Find out how to access and use DiRAC here. ®

Sign up to our NewsletterGet IT in your inbox daily


More from The Register

What's this under the Christmas tree? A gift-wrapped Mellanox, for Microsoft? Say it ain't so

Windows giant mulls gobbling up network kit maker, according to anon insiders

Imperial bringing in budget holograms to teach students

Suffice to say, it's not quite Tupac or Freddie Mercury

Mellanox plumps pipes on Bluefield controller to squeeze out 200Gb/s

That's right, count 'em. And 4 million IOPS to boot

Mellanox plumps up thanks to the storage world's hankering for high-speed Ethernet

Dell EMC and HPE help fuel record quarterly revenue

Mellanox flushes three directors at behest of activist investor

And agrees to do better or it'll have to hand over more board seats

Mellanox says it ships enough ports not to need Starboard

Spat with activist investor continues as companies feud over voting procedures

College PRIMOS prankster wreaks havoc with sysadmin manuals

Who, Me? Claims it was all for the greater good

Mellanox SoCs it to NVMe over Fabrics with BlueField platform

The JBOF made easier

Activist investor rages at Mellanox for dismissing Marvell's advances

Analysis Why won't you let us create value for shareholders?

They said yes, grins Dell Technologies: Expects to go public this month

Class V shareholders agree to sell or swap the stock