GitHub is down again

If you'd have asked me a few years ago if anything could be an existential threat to github's dominance in the tech community I'd have quickly said no.

If they don't get their ops house in order, this will go down as an all-time own goal in our industry.

Of course they're down while I'm trying to address a "High severity" security bug in Caddy but all I'm getting is a unicorn when loading the report.

(Actually there's 3 I'm currently working, but 2 are patched already, still closing the feedback loop though.)

I have a 2-hour window right now that is toddler free. I'm worried that the outage will delay the feedback loop with the reporter(s) into tomorrow and ultimately delay the patches.

I can't complain though -- GitHub sustains most of my livelihood so I can provide for my family through its Sponsors program, and I'm not a paying customer. (And yet, paying would not prevent the outage.) Overall I'm very grateful for GitHub.

You can literally watch GitHub explode bit by bit. Take a look at the GitHub Status History; it's hilarious: https://www.githubstatus.com/history.

Status page currently says the only issue is notification delays, but I have been getting a lot of Unicorn pages while trying to access PRs.

GitHub has had customer visible incidents large enough to warrant status page updates almost every day this year (https://www.githubstatus.com/history).

This should not be normal for any service, even at GitHub's size. There's a joke that your workday usually stops around 4pm, because that's when GitHub Actions goes down every day.

I wish someone inside the house cared to comment why the services barely stay up and what kinds of actions are they planning to do to fix this issue that's been going on years, but has definitely accelerated in the past year or so.

3 outages in 3 months straight according to their own status history. https://www.githubstatus.com/history

Screw GitHub, seriously. This unreliability is not acceptable. If I’m in a position where I can influence what code forge we use in future I will do everything in my power to steer away from GitHub.

Looks like AI replacement of engineering force in action.

What are good alternatives to GitHub for private repos + actions? I'm considering moving my company off of it because of reliablity issues.

They're overwhelmed with all the vibecoded apps people are pushing after watching the Super Bowl.

Meanwhile, Codeberg and Worktree are both online and humming along.

Codeberg gets hit by a fair few attacks every year, but they're doing pretty well, given their resources.

I am _really_ enjoying Worktree so far.

When I was a summer intern 10 years ago I remember there without fail always being a day where GitHub was down, ever summer. Good times.

Seems Microsoft goes downhill after all in AI.

Sorry, my fault. I tried to download a couple of CppCon presentations from their stash. Should have known better than to touch anything C++. ducks

I am able to access github.com at 140.82.112.3 no problem

I am able to access api.github.com at 20.205.243.168 no problem

No problem with githubusercontent.com either

GitHub is slowly turning into the Deutsche Bahn of git providers.

pretty clear that companies like microsoft are actually terrible at engineering, their core products were built 30 years ago. any changes now are generally extremely incremental and quickly rolled back with issue. trying to innovate at github shows just how bad they are.

Seems like MS copilot is vibe-ing it again ! Some other major cloud provider outages come to mind that never happened before the "vibe" area.

Monolith looking like a good now?

Oh! It's not my GitLab@Hetzner that's not working, it's GitHub. Just when I decided to opensource my project.

Is it really that much better than alternatives to justify these constant outages?

I guess Bill Gates has a virus.

Well its a day that ends in Y.

Github is down so often now, especially actions, I am not sure how so many companies are still relying on them.

I wonder what's the value of having a dedicated X (formerly Twitter) status account post 2023 when people without account will see a mix of entries from 2018, 2024, and 2020 in no particular order upon opening it. Is it just there so everyone can quickly share their post announcing they're back?

Looks like they've got a status page up now for PRs, separate from the earlier notifications one: https://www.githubstatus.com/incidents/smf24rvl67v9

Edit: Now acknowledging issues across GitHub as a whole, not just PRs.

Just remove all that copilot nonsense and focus on uptime... I would like to push some code.

The biggest thing tying my team to GitHub right now is that we use Graphite to manage stacked diffs, and as far as I can tell, Graphite doesn't support anything but GitHub. What other tools are people using for stacked-diff workflows (especially code review)?

Gerrit is the other option I'm aware of but it seems like it might require significant work to administer.

i was right ... https://medium.com/@patrick.szymkowiak/github-is-falling-apa...

It feels like GitHub's shift to these "AI writes code for you while you sleep!" features will appeal to a less technical crowd who lack awareness of the overall source code hosting and CI ecosystem and, combined with their operational incompetence of late (calling it how I see it), will see their dominance as the default source code solution for folks using it to maintain production software projects fade away.

Hopefully the hobbyists are willing to shell out for tokens as much as they expect.

It's a funny coincidence - I pushed a commit adding a link to an image in the README.md, opened the repo page, clicked on the said image, and got the unicorn page. The site did not load anymore after that.

Take it away from Microsoft. Not sure how this isn't an antitrust issue anyway.

On the plus side, it's git, so developers can at least get back to work without too much hassle as long as they don't need the CI/CD side of things immediately.

I wonder if GH charges for the runners during their downtime. Last week lot of them would retry multiple times and then fail.

List of company-friendly managed-host alternatives? SSO, auditing, user management, billing controls, etc?

I would love to pay Codeberg for managed hosting + support. GitLab is an ugly overcomplicated behemoth... Gitea offers "enterprise" plans but do they have all the needed corporate features? Bitbucket is a joke, never going back to that.

to be fair, i think usage has increased a lot because of coding agents and some things that worked well for now can't scale to the next 10x level.

Looks like they've got a status page up now for PRs, separate from the earlier notifications one: https://www.githubstatus.com/incidents/smf24rvl67v9

Edit: Now acknowledging issues across GitHub as a whole, not just PRs.

The saddest part to me is that their status update page and twitter are both out of date. I get a full 500 on github.com and yet all I see on their status page is an "incident with pull requests" and "copilot policy propagation delays."

https://www.githubstatus.com/

I don't know if it's related, but for the past week I've been getting pages cut off at some point, as if something closed the connection mid-transfer.

Today, when I was trying to see the contribution timeline of one project, it didn't render.

So what's the moneyline on all these outages being the result of vibe-coded LLM-as-software-engineer/LLM-as-platform-engineer executive cost cutting mandates?

Anyone have alternatives to recommend? We will be switching after this. Already moved to self-hosted action runners and we are early-stage so switching cost is fairly low.

Yeap, getting this for the last 20 minutes. Everything green on their status pages.

So, what're people's alt stack for replacing GitHub?

Azure infra rock solid as always.

I was wondering why my AUR packages won’t update, just my luck.

the incident has now expanded to include webhooks, git operations, actions, general page load + API requests, issues, and pull requests. they're effectively down hard.

hopefully its down all day. we need more incidents like this to happen for people to get a glimpse of the future.

They put too much AI in it bot enough engineering rigor

We replaced everything except the git part because of reliability issues. Pages…gone Actions…gone KB…gone. Tickets…gone.

Maybe they need to get more humans involved because GitHub is down at least once a week for a while now.

Damn, I was also trying to push and deploy a critical bug fix that was needed within minutes.