logoalt Hacker News

Ask HN: GitHub employees what's going on? Why?

198 pointsby shartsyesterday at 3:19 PM165 commentsview on HN

Pretty self explanatory. Could you folks shed some light on why these issues keep happening?

I’ve noticed most posts and replies are just people coming to their own conclusions based on whatever published data.

I think it would be helpful to get some actual, non-corporate/marketing information on the goings-on by those that actually see what’s happening on the inside.

Thanks


Comments

tristanjyesterday at 3:30 PM

Github is struggling because AI-boosted coding increased the number of commits 14x in the past year, and the pace is still accelerating. The site is struggling to keep up. Github's COO confirms it here: https://x.com/kdaigle/status/2040164759836778878

Platform activity is surging. There were 1 billion commits in 2025. As of three months ago, it was 275 million per week, on pace for 14 billion this year if growth remains linear (spoiler: it won't.)

20% of all GitHub accounts were created in the past 6 months https://x.com/kdaigle/status/2082604368399159542

show 14 replies
andyjohnson0today at 7:46 PM

Github used to be built on mysql / redis / ruby on rails / C / shell, running on dedicated hardware. Microsoft left it like that after they acquired the company.

Eventually, though, they decided to migrate the whole thing to Azure. And they were far enough through that to be basically committed... when AI coding started hitting them with much higher workloads.

I personally think the reliability problems are more to do with the reliability of the Azure migration. But both factors are likely relevant.

show 1 reply
ionwaketoday at 9:03 PM

I remember someone in that place telling me "i cant mention AI in my plans because they will laugh at me".

That was about 2 years ago. Im not kidding.

I dont know what the moral of the story is, but I found it weird at the time ( for added context - i was using AI back then about 10 hours a day, BUT I think sentiment on HN was "still" around the vibe of "you use AI to code without checking every line? I doubt your projects work" - but I had nothing better to do then wrestle with it and was surprised how I hadnt checked my code in weeks but stuff "worked". Its much better now and agents are accepted ofcourse but it sort of "snuck up" on people even in the tech community as recently as that.

I guess what Im saying is time is going quickly.

show 1 reply
Tbarlowtoday at 6:51 PM

14x commit growth in one year is brutal for any infrastructure. Scaling isn't just adding hardware.

alightsoultoday at 6:37 PM

Lovable only uses GitHub for storing projects. And requires people to provide their own GitHub account. Lots of vibe coders with no technical background are now having lovable push commits to GitHub

herpdyderpyesterday at 3:24 PM

non-helpful answer: the "Microsoft Acquires GitHub" line in this graph answers all questions https://damrnelson.github.io/github-historical-uptime/

show 7 replies
qrushyesterday at 4:02 PM

I got an "it is unacceptable" from their CPO on 8/7, and that they are "working around the clock on it".

https://x.com/mariorod1/status/2085800861469495465

I really think something deeper is going wrong there, and they're not being honest with their paying customers (and enterprises) about it.

solfoxyesterday at 3:27 PM

Many microsoft services are down/failing today, including sites hosted on Azure. I'm guessing it's a larger MS outage.

show 1 reply
mawadevtoday at 9:02 PM

We need a Github Remake without multiplayer from the new Github Studios

VCFundedGenYeryesterday at 3:54 PM

I think it's two things:

- GitHub attempting (and seemingly failing) to move to Azure infrastucture for its website backend

- AI generated code wrecking the site due to the volume of activities.

zeafoamrunyesterday at 3:47 PM

So you're asking github employees to violate their NDA?

show 1 reply
the_sleaze_today at 6:48 PM

Github moved to azure -> infrastructure problems

iJohnDoetoday at 7:02 PM

It's bad karma to sling mud about outages or problems. Cloudflare used to sling mud back in the day and then they went through some really bad outages afterwards.

llama052today at 8:31 PM

Github Status: Incident with GitHub.com

Aug 17, 21:15 UTC Resolved - On August 17, 2026, from 13:28–21:15 UTC (7h 47m),

GitHub.com experienced elevated errors and latency across Issues, Pull Requests, APIs, Actions, and Copilot. At peak, web/API error rates were approximately 20%, while archive and raw-content downloads reached approximately 50%. SAML/OIDC authentication, SCIM, and Team Sync were also affected, as well as Actions workflows in GHEC with Data Residency that depend on public workflow step definitions hosted on GitHub.com. Most services recovered by 16:36 UTC as our Central US datacenter recovered; Actions was degraded until approximately 18:03 UTC; and Copilot Token Service fully recovered by 21:02.

Some of the failing traffic was moved from Central US to Northern Virginia where it was served successfully until the network failure in Central US was debugged and resolved. Delayed replies to a single internal endpoint triggered a latent retry bug in VS Code that amplified traffic by approximately 10x and caused delayed recovery for the Copilot Token Service.

The immediate cause of the failure was network saturation on load balancers in Central US due to a new peak in traffic. Originally this was caused by an Istio sidecar pod reaching its concurrency limits and failing to auto scale correctly because of a misconfigured policy that watched host service but not sidecar limits. One failure cascaded to more and eventually four HAProxy nodes exhausted their flow limits, degrading the gateway auth path and causing widespread authentication latency and failures. The problem was worsened by optimistic retry logic which overloaded internal load balancers. Pausing HAProxy on those nodes simultaneously produced immediate broad recovery. The retry storm in Northern VA was fixed by 1) temporarily reducing gateway retry logic with a PR and 2) blocking inbound Copilot Token Service token requests at the load balancers with a 403, and then gradually ramping back up traffic per-site to allow callers to succeed. Residual Copilot authentication failures continued because client retry behavior amplified load: a failed token operation could generate many extra requests and enter a retry loop. Copilot Token Service traffic increased from a normal 7–9K RPS to 70–100K RPS. Reducing gateway authentication retries and blocking retry-triggering responses stabilized Copilot Token Service and completed recovery.

Complicating factors that impeded recovery included a number of scraping attacks on codeload endpoints.

To prevent recurrence, our follow-up actions include:

- Correcting autoscaling policies to account for service-mesh sidecar concurrency and capacity.

- Auditing Istio request, concurrency, and scaling limits across affected services.

- Reviewing retry limits and backoff behavior across gateways and clients.

- Addressing the VS Code retry behavior that amplified Copilot token traffic.

So basically bad code pushes that caused request amplification and then huge gaps in operational scaling and reliability standards. Oof.

show 1 reply
delducayesterday at 3:25 PM

AI + Microsoft = kabum

Alien1Beingyesterday at 3:56 PM

GitHub is struggling because it is owned by Microsoft.

toomuchtodoyesterday at 3:38 PM

GitHub Will Prioritize Migrating to Azure Over Feature Development - https://news.ycombinator.com/item?id=45517173 - October 2025 (63 comments)

HN Search: azure capacity - https://hn.algolia.com/?dateRange=all&page=0&prefix=true&que...

Azure Capacity Crunch Extends into 2026 Amid Data Center Constraints - https://windowsforum.com/threads/azure-capacity-crunch-exten... - October 9th, 2025

https://news.ycombinator.com/item?id=47427779

Taikhoom10today at 6:36 PM

and yet people do not leave, hence the power of switching costs. (Paying Customers).

https://s-1.vercel.app/posts/why-dropbox-is-a-obvious-pe-tar...

AregNoyatoday at 4:28 AM

i guess its because of the new cursor platform

plucyesterday at 3:33 PM

Microsoft. Azure. AI.

Pick one!

show 1 reply
rep_movsdyesterday at 4:21 PM

If you are doing 1000 commits a day, whats the point of git?

Does the AI ever look back at the shit trail it left behind?

Diffs are no longer diffs, they look like largescale delete and rewrite

show 1 reply
throwaway613746today at 5:34 PM

[dead]

platinumchartsyesterday at 3:49 PM

[dead]

mnmnmntoday at 6:45 PM

[dead]