Software

Devops

GitHub.com freezes up as techies race to fix dead data storage gear

TITSUP: Total Inability To Support Users' Pushes


GitHub's website remains broken after a data storage system failed hours ago.

Depending on where you are, you may have been working on some Sunday evening programming, or getting up to speed with work on a Monday morning, using resources on GitHub.com – and possibly failing miserably as a result of the outage.

From about 4pm US West Coast time on Sunday (2300 UTC), the website has been stuttering and spluttering. Specifically, the site is still up and serving pages – it's just intermittently serving out-of-date files, and ignoring submitted Gists, bug reports, pushes, and posts. Sometimes, it appears to be serving a read-only cache or older backup of itself, although some fresh code pushes are coming through onto the site.

Repairs

From the status page, it appears a data storage system died, forcing the platform's engineers to move the dot-com's files over to another box. In the meantime, some older versions of repos are being served to visitors and users.

"We're continuing to work on migrating a data storage system in order to restore access to GitHub.com," the team said just after 5pm PT, adding in the past few minutes: "We are continuing to repair a data storage system for GitHub.com. You may see inconsistent results during this process."

In this vulture's experience this evening, the backend git services are still working. It's just the website that's frozen. Some people have complained they can't log in, or that branches have gone missing, and so on.

Right now, we're seeing scores of complaints about the site being down on Twitter – including quite a few upset coders in Japan, where at time of writing is late Monday morning. Nice start to the week.

Confusion

Judging by the gripes, there's no obvious way to tell the site is borked, unless you look for the status page or realize something's up after the tenth attempt to submit work to the dot-com fails.

"Can you restore your service already?" software dev Saishav Agarwal pleaded to GitHub. "And please let users know when such issues are discovered! An email would be sufficient. Just wasted a precious hour."

Microsoft hopes to buy San Francisco-based GitHub for $7.5bn (£5.7bn). We'll update this story when, fingers crossed, GitHub repairs itself. ®

Updated to add at 0700 UTC

GitHub.com is still wonky, and will remain so for the next couple of hours, it seems. "We are currently in the later stages of a restore operation, with the aim of serving fully consistent data within the next 2 hours," Team GitHub said.

Updated to add at 1400 UTC

Five hours on, and users are still reporting inconsistencies, with several saying they are unable to log in or work. At 1318, GitHub staffers said: "We are validating the consistency of information across all data stores. Webhooks and Pages builds remain paused."

Final update

It's all fixed now, 24 hours later.

Send us news
69 Comments

Over 170K users caught up in poisoned Python package ruse

Supply chain attack targeted GitHub community of Top.gg Discord server

GitHub fixes pull request delay that derailed developers

Went down yesterday, too, longer and harder. Maybe we should call it GitFlub?

GitHub struggles to keep up with automated malicious forks

Cloned then compromised, bad repos are forked faster than they can be removed

What is GitHub Copilot Enterprise? You and your org just might find out firsthand

Big biz invited to like and subscribe, without fear of litigation

ANZ Bank test drives GitHub Copilot – and finds AI does give a helping hand

Expert Python programmers saw the most benefit

So, are we going to talk about how GitHub is an absolute boon for malware, or nah?

Microsoft says it's doing its best to crack down on crims

Cryptojackers steal AWS credentials from GitHub in 5 minutes

Researchers just scratching surface of their understanding of campaign dating back to 2020

If you want to fund open source code via Patreon with GitHub, well now you can

Exploited open source maintainers get broader payment pleading options

GitHub Copilot, Amazon Code Whisperer sometimes emit other people's API keys

AI dev assistants can be convinced to spill secrets learned during training

Researcher bags two-for-one deal on Linux bugs while probing GNOME component

One-click exploit could potentially affect most major distros

GitHub alienates developers by force feeding them AI recommendations

Decision to combine user-curated feed with algorithmic stuff leaves coders fuming

Microsoft and GitHub are still trying to derail Copilot code copyright legal fight

And so far, they might succeed: Where's the smoking gun?