Reg comments21

SourceForge staggers to feet after lengthy STORAGE FAIL outage

Warning: Dev services STILL COMATOSE until mid-week

Shaun of the Dead

Popular open source code-hosting repository SoureForge has been battling a significant outage for days and is slugglishly recovering from a lengthy Total Inability To Support Usual Performance (TITSUP) drama.

On Thursday, the site slipped into "disaster recovery mode" and since then it has been a torrid time for sysadmins scrabbling to restore the GitHub-like portal.

SourceForge, which along with news community Slashdot was swallowed by Dice Holdings in 2012, said in a blog post on Saturday that it was in the process of returning to normal service.

It shares infrastructure with Slashdot, which was also affected by the downtime. SourceForge reiterated that the sites had gone titsup due to a storage fault.

The Slashdot Media sites experienced an outage commencing last Thursday. We responded immediately and confirmed the issue was related to filesystem corruption on our storage platform.

This incident impacted all block devices on our Ceph cluster. We consulted with our storage vendor when forming our next steps. We have since been working 24×7 on data restoration, data validation, and service recovery.

Our response to date has been methodical and focused on safe restoration of data and service.

To enable this response we split our team in half, with one portion of the team working to expedite service restoration, and one portion of the team working on data validation and restoration.

While most of the site's projects have now been restored, SourceForge was yet to bring its developer data fully back to life.

"We’ll be bringing services back online as the validation of backing data is completed, and anticipate bringing additional services online through mid-week," SourceForge added.

"The data involved in our developer services is among the largest we house, and it takes time to perform filesystem checks and to restore data from backups. Using separate mounts, both steps are occurring concurrently to minimise the timetable for restore."

Welp. ®

Sign up to our Newsletter

Get IT in your inbox daily

Biting the hand that feeds IT © 1998–2017