alt Hacker NewsIncident with Github.com
Comments
I had a lot of goodwill for GitHub but I think today is the tipping point.
Looking at a unicorn page, I feel this lingering hope that it's transient (like it usually was in the old days) but my mind reassures me it's probably going to be a long full outage again.
The hope is dead.
I don't understand why Github hasn't solved this problem with pricing updates. My understanding is they are getting hammered with LLM generated code growing their traffic by over an order of magnitude. So why not rate limit non-paying users and charge for whatever scarce resource is being consumed that is causing them to constantly fall-over? This seems like a basic economics problem.
I wonder if you could replace the whole github issues feature with just a directory within each git repo named "Issues/". Within that, have another couple directories, "open/", and "closed/". Within "open", have a directory per issue, e.g. "such-and-such-does-not-work/". Within each issue directory, have "issue.txt", or "issue.md", which takes the place of the page for each issue on github. You could store other things in the issue directory which "issue.md" could reference, e.g. screen shots, logs, whatever. When an issue is closed, it gets git mv'ed to "closed/". You want to see a list of open issues sorted by time? Just "ls -ltr Issues/open". Want to see open issues related to the frakfrobber? "grep -i frakfrobber Issues/open/*". Add a comment to an issue? "vim Issues/open/my-issue/issue.md; git add !$; git commit -m 'Updated my-issue';" It would move the entire issues subsystem back under control of bare "git" and make it portable. Undoubtedly, I am not the first to think of such a thing.
The LLM code influx is not the problem. Microsoft mismanagement is: https://damrnelson.github.io/github-historical-uptime/
> Warning: Failed to download action 'https://codeload.github.com/'.
> Error: Response status code does not indicate success: 429 (Too Many Requests).
> Warning: Back off 19.714 seconds before retry.
> Warning: Failed to download action 'https://codeload.github.com/'.
> Error: Response status code does not indicate success: 502 (Bad Gateway).
> Warning: Back off 22.228 seconds before retry.
> Error: Response status code does not indicate success: 429 (Too Many Requests).
> Error: Failed to download archive 'https://codeload.github.com/' after 3 attempts.
Nice - even their own action runners are getting rate limited atm
I recall reading years ago that cloud services were expected to run with a reliability of 3 or 4 '9's and that if they didn't competing services would quickly overtake them in adoption. The industry was supposed to be that cut throat.
Has big tech reached a similar status like banks in that they are "too big to fail" i.e. when they do fail we all just look the other way and say: "well everyone else is out too". Didn't someone recently calculate that GitHub is running at 95%? For comparison the Irish Rail service which is not reliable has 80% of it's trains run on time.
This seems absurd and really challenges a lot of ideas I had about big tech and cloud infrastructure. GitHub seems to have remained the dominant player relative to GitLab etc.
Ah, so that's what the agent meant by summarizing a documentation page as "Unicorn!" I'd thought it was a very colorful hallucination (it's just GitHub's (unicorn-themed) failure page).
Gathered details on embedding request syntax from
Unicorn! · GitHub
To everyone who is angry: calm down. Github’s servers are constantly on fire as their usage increased something like 50x due to LLM sloppers pushing large amounts of trash code
Unless you have a emergency hotfix (you don’t), go hit the gym or walk outside.
If a work tool going down triggers you enough mentally to start angrily ranting online, it’s a sign you need to chill out and focus more on your health
What I want:
A self hosted version of GitHub that is a precise mirror of an actual GitHub repository. That is, PRs from GitHub appear on the selfhosted instance and my reviews from the self hosted instance appear on GitHub. That way I can work on my private instance when GitHub is down (or slow) and I can be assured my work still appears to everyone online.
Does this exist?
GitHub, Charles Schwab, Microsoft Teams, Copilot, New York State, AWS, Progressive, Azure, Microsoft Store, Cloudflare, ADP, Google Sheets, E-Trade, Fidelity... Did Mythos break out of it's container?
We're starting to have the conversation of "do we need to replace GitHub as our CI/CD pipeline?"
No idea if/when we decide to pull that bandaid but I think once we can't merge a hot fix into main because of GH being down, it'll be a big org push to replace it.
What's everyone else using? Self hosted Gitlab? Gitea?
The PSA is that you should always have a mirror—even if just a bare repo folks can pull from during downtime.
…& this is ignoring all the moral reason for avoiding US-based, publicly-traded, proprietary software for host largely free software. They also deny access to users due to US sanctions, privacy-conscious users won’t want to create accounts/agree to ToS, & rubbing shoulders with other corpos your project can be taken down if it upsets the corpos don’t like it (see youtube-dl, Switch emulators, etc.).
> We are seeing an approximate 20% error rate across numerous experiences including Pull Requests, Issues, and others. Investigations are currently under way and we will be posting updates as they become available
Well it's 100% down for me, so does this mean 20% of all users are fully down?
I'm self-hosting Gitea and Woodpecker CI for least 6 years, alongside my own docker and pypi registries. Adding x86 and ARM runners, it costs me circa $30/month to run the whole thing, and I am reasonably sure that I could make the whole thing scale at a lower cost per user than what Github charges.
I am seriously, 100%, contemplating the idea of offering consulting for migrating companies and projects out of Github. Given that I am fully employed, I'd only be able to do this on the weekends. Could I get a show of hands to get an idea of potential customers?
Here's an article from Oct 2025:
"Microsoft’s data-center shortages to persist longer than expected - Bloomberg"
https://ca.finance.yahoo.com/news/microsoft-data-center-shor...
Perhaps relevant tho. I would imagine if they could simply throw more hardware at it they would. And I wouldn't imagine inability to throw hardware at it would be due to software scaling issues. So perhaps it is simply lack of available hardware?
Funny enough, I received a marketing email from GitHub during the outage with the following subject: "What if every pull request you started actually shipped"
So how’s using Azure to power GitHub going?
Seriously. It seems like there’s an issue every other week. It would historically be normal to provide post mortems, and maybe I’ve missed them but seems like there are almost weekly issues and no follow up into the core reasons why.
(Umpteenth time:) GitHub is not about just publishing source code; with git, it's trivial out of the box.
GitHub's value is in the PR process, in CI/CD, releases, the issue tracker, and an identity system propping all of that. This is the thing to replace, and it's quite a bit harder.
Funny thing about the status page is that you can't find any record of last Thursday's outage. It's all green across every service.
If you really need to merge and deploy, I've been having some luck getting things moving with the gh cli:
gh pr review <xx> --approve
gh pr merge <xx> --squash --delete-branch
It's getting to where we should just post when it's working fine, to reduce noise.
I really wish I could move to a different provider, but GitHub is just so damn practical. Downtimes such as this should at least be compensated, providing an actual financial incentive to improve their services, instead of slapping AI on yet another product.
This is getting ridiculous. Issues, PRs, actions... nothing's working
Can anyone shed some light why GitHub is having so many problems lately?
What is the best Github alternative that has something like Github Actions? What is the easiest to switch to? I do not want to switch but it is having too much of an impact on me.
It is time to give up on Github. This is just too much.
Issues seems to be completely broken, just getting unicorns.
Interestingly the API still works for creating an issue (but webhooks weren't fired).
I can imagine the massive growth they're in right now, old assumptions are breaking down, many things need to be rethought. That's why personally i don't hate them with all of this saga. But they need to work faster in terms of product decisions and technology.
Could GitHub be affecting other sites, or maybe this is something else?
Did Anthropic or OpenAI fail to sandbox their models again?
All of my code repos are on gitea (personal and professional). The instance is not exposed to the internet (wireguard access). It's fantastic honestly, simple and just works.
What are some good SaaS alternatives to GitHub for CI (that aren't GitLab)? Is anyone doing anything particularly novel in this space?
At this point, they should notify us on days that it's up
What is really interesting is that no insider engineer has leaked anything. What we got was essentially "LLM" hitting GitHub really hard. But there is no retro or maybe i missed it.
GitHub is down days of that week that end with day these days. If I were a paying customer I'd be cancelling soon. GitHub is sort of foundational at this point and when its down nearly every day of the week it cannot be foundational.
How did Github become so famous for outages? It is almost like a routine. I haven't used other code repo hosting / CI/CD pipelines platform for last decade, so maybe it's just the nature of this business, I don't know.
Someone tried loading the Bun Zig-to-Rust PR again.
So glad I moved my OSS org mostly over to a self-hosted Forgejo + Woodpecker CI. I get that we are operating on a completely different scale but I swear the GitHub outages are far too regular than they ought to be.
Possible if this trend continues we might have the first actual zero 9s availability.
Has Github Enterprise Server been isolated from these issues? Presumably it has because it's self hosted
Never quite understood why large enterprises use github as core infrastructure.
Sure use it to collab externally, use it if you can't justify running your own infrastructure, but why are large companies choosing to place code externally - which is one secret leak away from compromise?
Awwww maaaan. I was just reading an upstream patch on the erpnext framework. I was going to do their copy the pr to patch my erpnext v15
The GitHub failure unicorn image always brings me back to the good old days of trying out new Android ROMs. That unicorn used to be the logo of the Android Open Kang Project, which used to be my favorite ROM.
I wonder how it made it onto GitHub.
Why haven't we all moved off GitHub yet?
Or let me ask a more useful question: if we were to move, then to where? Ideally somewhere new (not Gitlab)
A self-hosted git-based issue tracker like Epiq [1] seems more relevant by each day going forward.
You know, I think Fossil with its integrated issue management right in the repo is actually a perfect match for the LLM era. Maybe today is the day I migrate to using a Fossil forge with GitHub becoming increasing untenable.
Anyone else using Fossil?
1) Issues showed unicorn. Reloading page.
2) Issues showed Not found.
Anyway, GitHub Actions for building new release has progress. Good enough.
If someone needs lightweight self hosted git server with ci pipelines via general programming languages - consider DSCI - http://deadsimpleci.sparrowhub.io
Anyone remember the end of the .com era? This feels early similar.
I remember a few years back when there was 2 major outages per month.
Thankfully this is encouraging me to stop working overtime for now and enjoy the rest of my night
I've actually started loving GitHub down days because it means I get to spend time migrating away from them! (moving to self hosted Gitea)
This is starting to feel like Twitter. For a variety of reasons, we had a centralized place where everyone of a particular set of persuasions could connect, and this had outsize benefits for the community as a whole. That place is becoming untenable, and with the loss of goodwill and stability, the community’s started to dissipate. But there isn’t one obvious transition candidate, so the diaspora is finding itself spread across a bunch of disparate places and services. The stated need they satisfy (source control with a web interface and some technical features, like pull requests with reviews) will be fulfilled. But those emergent features like a core community and default expectation of where you can find someone will fade. And that is a very real loss for all of us.