Brit uni builds its own supercomputer from secondhand parts

She may not look like much, but she's got it where it counts, kid

By Chris Mellor

Posted in HPC, 4th August 2017 11:41 GMT

Durham University has built itself a secondhand supercomputer from recycled parts and beefed up its contribution to DiRAC (distributed research utilising advanced computing), the integrated facility for theoretical modelling and HPC-based research in particle physics, astronomy and cosmology.

The Institute for Cosmological Computing (ICC) at Durham, in northeast England, runs a COSMA5 system as its DiRAC contribution.

There are five DiRAC installations in the UK, which is a world leader in these HPC fields:

The Durham cluster listed above is a COSMA5 system, which features 420 IBM iDataPlex dx360 M4 servers with a 6m720 2.6 GHz Intel Sandy Bridge Xeon E5-2670 CPU cores. There is 53.76TB of DDR3 RAM and Mellanox FDR10 Infiniband in a 2:1 blocking configuration.

It has 2.5PB of DDN storage with two SD12K controllers configured in fully redundant mode. It's served by six GPFS servers connected into the controllers over full FDR and using RDMA over the FDR10 network into the compute cluster. COSMA5 uses the GPFS file system with LSF as its job scheduler.

The ICC and DiRAC needed to strengthen this system and found that the Hartree Centre at Daresbury had a supercomputer it needed rid of. This HPC system was installed in April 2012 but had to go because Daresbury had newer kit.

Durham had a machine room with power and cooling that could take it. Even better, its configuration was remarkably similar to COSMA5.

So HPC, storage and data analytics integrator OCF, and server relocation and data centre migration specialist Technimove dismantled, transported, and rebuilt the machine at the ICC. The whole exercise was funded by the Science and Technology Facilities Council.

COSMA6 arrived at Durham in April 2016, and was installed and tested at the ICC. It now extends Durham's DiRAC system as part of DiRAC 2.5.

COSMA6 has:

The Lustre filesystem and SLURM are used for its job submission system.

COSMA6 racks

Lydia Heck, ICC technical director, said: "While it was quite an effort to bring it to its current state, as it is the same architecture and the same network layout as our previous system, we expect this to run very well."

Durham now has both COSMA5 (6,500 cores) and COSMA6 (8,000 cores) contributing to DiRAC and available for researchers.

Find out how to access and use DiRAC here. ®

Sign up to our NewsletterGet IT in your inbox daily

56 Comments

More from The Register

Activist investor rages at Mellanox for dismissing Marvell's advances

Analysis Why won't you let us create value for shareholders?

Mellanox SoCs it to NVMe over Fabrics with BlueField platform

The JBOF made easier

Mellanox NICs Xilinx FPGA to save backplane slots and CPU cycles

And it's not just about bonkers Bitcoin mining rigs

Dell EMC adds Skylake grunt to supercomputing workhorse server

New Tesla GPU sends single precision performance past 62 TFLOPS

Back to ASICs: Mellanox pumps up Ethernet speed to 400Gbps

200GbitE and 400GbitE ASIC-powered switches

Dell sell-off saga gets weird: Subsidiary VMware may buy parent in 'reverse merger'

Buy-out would let Big Mike swerve IPO headaches

Mangstor, Mellanox flash rig crowned 'fastest in the lab'... for RAID-0

Reviewer says NVMeF-based MySQL cluster is darned quick

Manchester college swaps out disk for rackful of hybrid flash

Single hybrid array replaces 8 disk boxes

Perv raided college girls' online accounts for nude snaps – by cracking their security questions

Personal info obtained to pull off 1,400 password resets. Now he's behind bars

Dell intranet post said VMware slurp disclosure was mere paperwork

Staff told there’s nothing to see here, get back to work and stop worrying about our debt