logoalt Hacker News

Incident with multple GitHub services

170 pointsby bwannasektoday at 4:21 PM87 commentsview on HN

Comments

embedding-shapetoday at 4:59 PM

I've been on a somewhat binge to move a bunch of stuff to self-hosting at home. Yesterday I finally completed my self-hosted Forgejo instance at home, together with Linux, Windows (via VM) and macOS (via Mac Mini) runners/workers for CI/CD, so everything finally lives in-house (literally), instead of all source code + Actions being on GitHub but the infrastructure actually living locally.

This is probably the first time I felt vindicated with my self-hosting move literally the day after I finished the migration, very pleasant feeling. Usually it takes a month or two before I get here.

show 10 replies
LorenDBtoday at 5:09 PM

https://mrshu.github.io/github-statuses/ says they are down to 88.15% uptime. Even when you consider uptime of individual components, their best is 99.78%, so two nines.

show 2 replies
hedayettoday at 10:29 PM

Is Github losing any significant business from all these outages?

Curious because for a long time we as an industry maintained that reliability and brand value are business critical; but seems like they are cared very little now a days.

Happy to be corrected about my perception too.

show 1 reply
deferredgranttoday at 10:23 PM

These outages are also a good test of how much local resilience teams actually built. My guess is most shops are much more dependent on GitHub than they like to admit.

frereubutoday at 8:40 PM

We have pretty basic needs - git repos + actions - and a bit of downtime here and there doesn't really affect us too much because we're not constantly committing and deploying, but even we're looking around for alternatives now.

Also, looks like people might be pummelling the SourceHut servers looking for an alternative: https://sr.ht/ is down. (Edit: was down when I wrote that, back up now).

show 1 reply
cjonastoday at 4:50 PM

It would be wild if they dropped below the "two 9's" metric. I think they would need an additional ~16hr of outage in the 90 day rolling period.

show 2 replies
Groxxtoday at 9:35 PM

So... three incidents today, all of them ~1h or longer, and everything's green for the day with "no recorded downtime".

These don't really look any different than past incidents which have red bars on their respective days, except maybe that those tended to be several hours.

What do the green bars even mean? Are they changed to non-green retroactively if people complain enough or something? So far as I can tell, literally none of the previous green days have any incident shown in the mouse-over, but there are multiple for today only, so I kinda have to assume the mouse-overs are conveniently "forgotten" or all incidents become non-green and they just don't bother informing anyone on the same day. Either seems intentionally misleading.

agartnertoday at 10:01 PM

There was another really bad incident today: https://www.githubstatus.com/incidents/zsg1lk7w13cf

> We have resolved a regression present when using merge queue with either squash merges or rebases. If you use merge queue in this configuration, some pull requests may have been merged incorrectly between 2026-04-23 16:05-20:43 UTC.

We had ~8 commits get entirely reverted on our default branch during this time. I've never seen a github incident quite this bad.

show 2 replies
dankobgdtoday at 5:01 PM

Don't worry, status page says that it's 100% working - green color, all good. even though i can't access a static page

delusionaltoday at 9:00 PM

Once that 10x developer velocity from AI kicks in, I'm sure github stability improves. Did you know AI finally makes it economical to fix all the little bugs?

argeetoday at 4:47 PM

I moved to Gitlab a while ago. It's a whole new level of freedom not having to pay for self-hosted CI runners.

jonnonztoday at 7:19 PM

Well I suppose they are finding out if you lay off too many people the IP of how the system works goes out the door with them.

bakiestoday at 5:25 PM

I definitely have better uptime hosting my own gitea instance. It's faster too. It's basically a knock off GitHub. Plus with privacy concerns, I'm just happier overall. Easy setup, all I did was deploy the helm chart.

CamouflagedKiwitoday at 5:40 PM

I wondered. We'd seen for most of today that Actions were slow to trigger, I had at least one that was just missed, it felt like something was definitely off but the status was green all day until this.

jasoncartwrighttoday at 5:23 PM

Just cancelled my GitHub Copilot Pro+ year subscription. Removal of Opus 4.6 stung, but the repeated continued downtime makes it unusable for me. Very disappointed.

No fuss instant refund of my unused subscription (£160) appreciated.

show 2 replies
AnkerSkallebanktoday at 4:51 PM

Some of my jobs are completing, some are failing. Seems to be random. Kind of wish they would just fail outright, instead of running for 10 minutes and then failing.

fishgoesblubtoday at 4:50 PM

At this point it'll be better to have alerts for when GitHub is online, rather than offline.

surya2006today at 5:04 PM

what are the good alternatives available for github i find some alternative but as long as widely people use github i cant use other service right like i cant share my alternative to other developer and force him to use this for me. so i feel like i locked in even i want to move i can't

show 5 replies
tomjen3today at 5:22 PM

At this point it should almost be news when it works.

ChrisArchitecttoday at 4:58 PM

Multple 9s

buildbottoday at 5:08 PM

Anyone also seeing Active Directory/Entra issues?

supakeentoday at 4:42 PM

I mean; this is the normal mode of operation for GitHub at this point.

show 1 reply
surya2006today at 5:05 PM

even vercel also have more downtime nowadays

nhhvhytoday at 7:00 PM

If the day ends in Y…

throwatdem12311today at 5:08 PM

Seems like they just can’t deal with the absolute deluge of AI vomit being uploaded every day.

Good riddance I hope it completely destroys them.

show 1 reply
poplarsoltoday at 7:20 PM

Azure webapp deploys are also trash right now. Microsoft needs to stop slathering h1b copilot slop and get basic things like Windows patches working.

linhnstoday at 4:57 PM

Business as usual.

0xbadcafebeetoday at 4:57 PM

I am this > < close to just running Gogs or Forgejo on some Hetzner boxes, quit my job, charge people for access. Why aren't there like 10 startups doing this yet? Please? I want to give you my money. Just give me a git host that doesn't suck. (All the current ones suck)

show 2 replies
shevy-javatoday at 5:05 PM

Microsoft again.

I think it is time that Microsoft lets go of GitHub. They are handling it too poorly.

josefritzisheretoday at 4:51 PM

Microslop is destroying Github

ossa-matoday at 4:55 PM

Seems like outages are increasingly more frequent nowadays. Obviously, this is not the best state of affairs, and developers should not be limited by services. In the meantime I've been experimenting with building third spaces for people to chill while they wait for the services they are dependent on to go back up.

The first one I've built is a little ASCII hangout for Claude @ https://clawdpenguin.com but threads like this make me want to build it for Github too.