alt Hacker NewsMicrosoft outage gives customers a crash course in dependence on Redmond
Comments
I think every N days a chaos monkey event should happen for every business to help with this kind of issue. If a SaaS takes your business offline, you aren't resilient.
That ship sailed 20 years ago when everybody stampeded into cloud computing / SaaS ago and it became heresy to suggest on-prem had benefits. "Redmond" is a symptom but the problem is "live by the cloud, die by the cloud."
When you're a monopoly, you don't have to worry about performance. Nobody is going anywhere, they'll just wait.
[flagged]
Back in 2011 various email worms decimated Exchange mail servers. It cost billions of dollars in productivity and the company I was working for was on lotus notes but migrating to Exchange. Did taking down systems all around the world stop them from migrating? Of course not. The Microsoft Stockholm syndrome is real.
The postmortem on this will be interesting. A few days prior Microsoft said several versions of on premise Exchange are permanently vulnerable.
Maybe AI scanning found these Exchange exploits and after laying off so many developers, they vibe coded an exploit fix and deployed it. But behind the scenes it is all just still Exchange which if anyone has experience managing, its a complex nightmare of code going back decades. With all sorts of Active Directory requirements and everything else. The mail database system it uses, and probably still used in Office 365 (Office 363 today?) is also a nightmare.
Just speculation though
Microsoft says "while this took longer..." [1] but they should have said "while this is taking longer" because the outage isn't fixed yet. What kind of outage takes 2 days to fix?
[1] https://x.com/Microsoft/status/2121282013600907325
"The incident was introduced by an issue during an earlier network capacity upgrade that surfaced following a recent network change. We have addressed the underlying issue and continue to restore service to remaining customers and regions.
Given the scale of demand, we restored services gradually, validating stability at each stage. While this took longer, it helped ensure a stable restoration."
Microsoft: "I've Fallen and I Can't Get Up"
All the CEO's AI and All the CEO's Men / Couldn't Put Microsoft Together Again
It’s pretty crazy how much Microsoft can get away with. Almost a government level of getawayability.
naa .. it's totally fine. The only thing that will move consciences will be data loss/leak at scale. But even in that case I have some doubts ...
And, if eventually a big bunch of CTOs starts considering going back so self-hosting/managing people will pop up everywhere saying "even if <big corp> sucks, you cannot do better" or "it will cost you more" or "what about the peak usage".
It's incredible how we ended up paying much more, giving voluntarily all our data, and getting a shitty service.
Not really. Every time a giant cloud provider goes down it reminds us a that it used to be our problem.
“Aren’t you glad it’s them and not us?”
I wonder if it'll be a more useful course than the one we had when Crowdstrike broke everything, or the many times when AWS US-East-1 has fallen over taking down many popular services.
The lesson most people learn from this sort of outage is that things bounce back afterwards, and the risk is minimal enough that a day out of every few years isn't enough of a problem to pay more.