Feeds

Storage glitches fell Australian supercomputers

DDN and SGI find fix after faulty firmware fingered for Fornax fail

Beginner's guide to SSL certificates

Supercomputers at two Australian research organisations have experienced substantial downtime after glitches hit their storage area networks.

West Australia’s iVEC experienced an outage, detailed here, that saw its Data Direct Networks (DDN) array and the 1152-core Fornax machine unavailable for around four days.

iVEC’s incident report for the outage says the machine was shut down on December 17th to allow the update of firmware in a “SGI DDN IS16000 storage unit.” The update produced symptoms including disks going missing from storage pools and loss of availability of some data. Instructions from the vendors concerned saw the arrays go through a lengthy “force verify process” followed by a RAID rebuild in some cases.

The rebuild started on December 18th and finished on the 20th, when it became possible to restart Fornax.

eResearch South Australia’s Tizard and Corvus supers, which offer 40 TFlops and 6 TFlops apiece, went down on December 27th and were coaxed back to life on January 3rd, a period covering three business days.

The organisation described the outage as follows on its (since updated) system maintenance page:

“During the holiday break our primary storage system locked up causing the Tizard and Corvus HPC systems to become unresponsive. At the moment there are a large number of orphaned jobs on the compute nodes of both HPC systems due to the outage.”

The culprit for the outage was not the Dell Compellent array that serves as shared NAS for the two supers, but a Dell server that sits at a critical juncture in the InfiniBand network linking the supers to their storage. Simon Brennan, a computer systems specialist at eResearch SA, told The Reg the server was overwhelmed by the quantity of data coursing through it, crashed and was restored once staff resumed from their Christmas break.

Brennan said the server at the heart of the issue was a beta, and will shortly be completely rebuilt. ®

Security for virtualized datacentres

More from The Register

next story
It's Big, it's Blue... it's simply FABLESS! IBM's chip-free future
Or why the reversal of globalisation ain't gonna 'appen
'Hmm, why CAN'T I run a water pipe through that rack of media servers?'
Leaving Las Vegas for Armenia kludging and Dubai dune bashing
Facebook slurps 'paste sites' for STOLEN passwords, sprinkles on hash and salt
Zuck's ad empire DOESN'T see details in plain text. Phew!
CAGE MATCH: Microsoft, Dell open co-located bit barns in Oz
Whole new species of XaaS spawning in the antipodes
Microsoft and Dell’s cloud in a box: Instant Azure for the data centre
A less painful way to run Microsoft’s private cloud
AWS pulls desktop-as-a-service from the PC
Support for PCoIP protocol means zero clients can run cloudy desktops
prev story

Whitepapers

Choosing cloud Backup services
Demystify how you can address your data protection needs in your small- to medium-sized business and select the best online backup service to meet your needs.
Forging a new future with identity relationship management
Learn about ForgeRock's next generation IRM platform and how it is designed to empower CEOS's and enterprises to engage with consumers.
Security for virtualized datacentres
Legacy security solutions are inefficient due to the architectural differences between physical and virtual environments.
Reg Reader Research: SaaS based Email and Office Productivity Tools
Read this Reg reader report which provides advice and guidance for SMBs towards the use of SaaS based email and Office productivity tools.
Storage capacity and performance optimization at Mizuno USA
Mizuno USA turn to Tegile storage technology to solve both their SAN and backup issues.