I’m firmly in the camp of “something stinks at GitHub” but

> It is unbelivable that aload of 2.8b commits was totally fine, and a load of 2.9b was a sitewide outage

In my experience, there are hard thresholds that get passed that expose hidden bottlenecks like this. A previous system I worked on we had absolutely loads of headroom by all of our measured metrics, but one day we filled a cache because the value hadn’t been tweaked in recent memory. Plenty of space on disk and in memory, but all of a sudden we went from a very high cache hit rate to a very low cache hit rate, and everything ground to a halt.