Data Centre

HPC

Brit uni builds its own supercomputer from secondhand parts

She may not look like much, but she's got it where it counts, kid

By Chris Mellor

56 SHARE

Durham University has built itself a secondhand supercomputer from recycled parts and beefed up its contribution to DiRAC (distributed research utilising advanced computing), the integrated facility for theoretical modelling and HPC-based research in particle physics, astronomy and cosmology.

The Institute for Cosmological Computing (ICC) at Durham, in northeast England, runs a COSMA5 system as its DiRAC contribution.

There are five DiRAC installations in the UK, which is a world leader in these HPC fields:

The Durham cluster listed above is a COSMA5 system, which features 420 IBM iDataPlex dx360 M4 servers with a 6m720 2.6 GHz Intel Sandy Bridge Xeon E5-2670 CPU cores. There is 53.76TB of DDR3 RAM and Mellanox FDR10 Infiniband in a 2:1 blocking configuration.

It has 2.5PB of DDN storage with two SD12K controllers configured in fully redundant mode. It's served by six GPFS servers connected into the controllers over full FDR and using RDMA over the FDR10 network into the compute cluster. COSMA5 uses the GPFS file system with LSF as its job scheduler.

The ICC and DiRAC needed to strengthen this system and found that the Hartree Centre at Daresbury had a supercomputer it needed rid of. This HPC system was installed in April 2012 but had to go because Daresbury had newer kit.

Durham had a machine room with power and cooling that could take it. Even better, its configuration was remarkably similar to COSMA5.

So HPC, storage and data analytics integrator OCF, and server relocation and data centre migration specialist Technimove dismantled, transported, and rebuilt the machine at the ICC. The whole exercise was funded by the Science and Technology Facilities Council.

COSMA6 arrived at Durham in April 2016, and was installed and tested at the ICC. It now extends Durham's DiRAC system as part of DiRAC 2.5.

COSMA6 has:

The Lustre filesystem and SLURM are used for its job submission system.

COSMA6 racks

Lydia Heck, ICC technical director, said: "While it was quite an effort to bring it to its current state, as it is the same architecture and the same network layout as our previous system, we expect this to run very well."

Durham now has both COSMA5 (6,500 cores) and COSMA6 (8,000 cores) contributing to DiRAC and available for researchers.

Find out how to access and use DiRAC here. ®

Sign up to our NewsletterGet IT in your inbox daily

56 Comments

More from The Register

Mellanox plumps pipes on Bluefield controller to squeeze out 200Gb/s

That's right, count 'em. And 4 million IOPS to boot

Mellanox plumps up thanks to the storage world's hankering for high-speed Ethernet

Dell EMC and HPE help fuel record quarterly revenue

Mellanox flushes three directors at behest of activist investor

And agrees to do better or it'll have to hand over more board seats

Mellanox says it ships enough ports not to need Starboard

Spat with activist investor continues as companies feud over voting procedures

Activist investor rages at Mellanox for dismissing Marvell's advances

Analysis Why won't you let us create value for shareholders?

Mellanox SoCs it to NVMe over Fabrics with BlueField platform

The JBOF made easier

Mellanox NICs Xilinx FPGA to save backplane slots and CPU cycles

And it's not just about bonkers Bitcoin mining rigs

Dell's hokey cokey IPO takes new turn – VMware in, VMware out....

Investor roadshow delayed as Mick D considers alternative plan

Dell EMC adds Skylake grunt to supercomputing workhorse server

New Tesla GPU sends single precision performance past 62 TFLOPS

Dell Tech: We have a plan B for bog standard IPO

Contingency plan in place if VMware investors don't like reverse merger proposal