alt Hacker NewsSomething I've been thinking about with the last few major outages is that as there is more and more consolidation (fewer large companies running a higher share of the world's infrastructure and critical services) is that we become increasingly exposed to massive disruption like this. Would some sort of mandated maximum occupancy per instance resolve this? So Outlook.com is allowed to support 100m accounts per instance. Then you are required to spin up Outlook2.com. You are not allowed to have any shared infrastructure. Yes it's wasteful and expensive, but these are companies with billions in revenue, so that's the price you pay. It just seems that keeping these huge, incredibly complex services running is turning in to an impossible task. Very much a hot take so interested in the opinions of peope who have worked at this scale.
Replies
They could (and should, and clearly don’t) architect the services like this internally. There will always be some single-points-of-failure, but minimizing those should be a top priority. Just from reading their status updates it’s clear most everything is sharing way too much in terms of infrastructure and configuration.
But even this requires someone high up in the organization to have the insight that reliability requires some level of “inefficiency” — ie retaining certain highly paid staff, duplication of effort, over provisioning of resources, etc.
The irony is that the execs who could make those decisions are convinced that LLM tools obviate those needs, and shift all spare resources in that direction. But without heavy constraints from experienced humans, LLM tools will only multiply the complexity at the root cause of these sorts of outages.
I am positive there are bare-metal, no shared tenancy, customers of MS already. Unclear if they are down too. While yes, these companies are massive, not being allowed to share infrastructure defeats the entire revenue-purpose of offering a cloud. The hardware in these servers is unbelievably expensive. You'd go back to everyone having an IT department and a few racks of servers for hardware configurations that meet their specific needs (which I'm totally okay with, and even rooting for!)
I'm an outsider who used to work at a different cloud provider, so take this with a grain of salt, but this doesn't smell like a limits and provisioning problem, per se. It's not necessarily like they just had 1 too many emails tip the scales for the entire world. Until today stuff was running okay. This was a monitoring and fail-over problem, at a comically tragic scale - a cascade failure. Architecturally they obviously did not design for a world where they could damage control failure at this scale. Failure should be expected and designed for at literally every layer. They cheaped-out on careful design in order to play catch-up building a public cloud offering and here we are.