GitHub.com freezes up as techies race to fix dead data storage gear

TITSUP: Total Inability To Support Users' Pushes

GitHub's website remains broken after a data storage system failed hours ago.

Depending on where you are, you may have been working on some Sunday evening programming, or getting up to speed with work on a Monday morning, using resources on GitHub.com – and possibly failing miserably as a result of the outage.

From about 4pm US West Coast time on Sunday (2300 UTC), the website has been stuttering and spluttering. Specifically, the site is still up and serving pages – it's just intermittently serving out-of-date files, and ignoring submitted Gists, bug reports, pushes, and posts. Sometimes, it appears to be serving a read-only cache or older backup of itself, although some fresh code pushes are coming through onto the site.

Repairs

From the status page, it appears a data storage system died, forcing the platform's engineers to move the dot-com's files over to another box. In the meantime, some older versions of repos are being served to visitors and users.

"We're continuing to work on migrating a data storage system in order to restore access to GitHub.com," the team said just after 5pm PT, adding in the past few minutes: "We are continuing to repair a data storage system for GitHub.com. You may see inconsistent results during this process."

In this vulture's experience this evening, the backend git services are still working. It's just the website that's frozen. Some people have complained they can't log in, or that branches have gone missing, and so on.

Right now, we're seeing scores of complaints about the site being down on Twitter – including quite a few upset coders in Japan, where at time of writing is late Monday morning. Nice start to the week.

Confusion

Judging by the gripes, there's no obvious way to tell the site is borked, unless you look for the status page or realize something's up after the tenth attempt to submit work to the dot-com fails.

"Can you restore your service already?" software dev Saishav Agarwal pleaded to GitHub. "And please let users know when such issues are discovered! An email would be sufficient. Just wasted a precious hour."

Microsoft hopes to buy San Francisco-based GitHub for $7.5bn (£5.7bn). We'll update this story when, fingers crossed, GitHub repairs itself. ®

Updated to add at 0700 UTC

GitHub.com is still wonky, and will remain so for the next couple of hours, it seems. "We are currently in the later stages of a restore operation, with the aim of serving fully consistent data within the next 2 hours," Team GitHub said.

Updated to add at 1400 UTC

Five hours on, and users are still reporting inconsistencies, with several saying they are unable to log in or work. At 1318, GitHub staffers said: "We are validating the consistency of information across all data stores. Webhooks and Pages builds remain paused."

Final update

It's all fixed now, 24 hours later.




Biting the hand that feeds IT © 1998–2018