GitHub.com freezes up as techies race to fix dead data storage gear
TITSUP: Total Inability To Support Users' Pushes
GitHub's website remains broken after a data storage system failed hours ago.
Depending on where you are, you may have been working on some Sunday evening programming, or getting up to speed with work on a Monday morning, using resources on GitHub.com – and possibly failing miserably as a result of the outage.
From about 4pm US West Coast time on Sunday (2300 UTC), the website has been stuttering and spluttering. Specifically, the site is still up and serving pages – it's just intermittently serving out-of-date files, and ignoring submitted Gists, bug reports, pushes, and posts. Sometimes, it appears to be serving a read-only cache or older backup of itself, although some fresh code pushes are coming through onto the site.
From the status page, it appears a data storage system died, forcing the platform's engineers to move the dot-com's files over to another box. In the meantime, some older versions of repos are being served to visitors and users.
"We're continuing to work on migrating a data storage system in order to restore access to GitHub.com," the team said just after 5pm PT, adding in the past few minutes: "We are continuing to repair a data storage system for GitHub.com. You may see inconsistent results during this process."
In this vulture's experience this evening, the backend git services are still working. It's just the website that's frozen. Some people have complained they can't log in, or that branches have gone missing, and so on.
Right now, we're seeing scores of complaints about the site being down on Twitter – including quite a few upset coders in Japan, where at time of writing is late Monday morning. Nice start to the week.
Judging by the gripes, there's no obvious way to tell the site is borked, unless you look for the status page or realize something's up after the tenth attempt to submit work to the dot-com fails.
"Can you restore your service already?" software dev Saishav Agarwal pleaded to GitHub. "And please let users know when such issues are discovered! An email would be sufficient. Just wasted a precious hour."
Microsoft hopes to buy San Francisco-based GitHub for $7.5bn (£5.7bn). We'll update this story when, fingers crossed, GitHub repairs itself. ®
Updated to add at 0700 UTC
GitHub.com is still wonky, and will remain so for the next couple of hours, it seems. "We are currently in the later stages of a restore operation, with the aim of serving fully consistent data within the next 2 hours," Team GitHub said.
Updated to add at 1400 UTC
Five hours on, and users are still reporting inconsistencies, with several saying they are unable to log in or work. At 1318, GitHub staffers said: "We are validating the consistency of information across all data stores. Webhooks and Pages builds remain paused."
It's all fixed now, 24 hours later.