Microsoft says attempts to fix global outage not going 'as expected'

FYI, if your business uses Microsoft 365, you are eligible to having 50% of your monthly fee waived off as per MS online services SLA.

It's been 18 hours since the start of the outage. If service isn't restored in the next 18 hours (36hrs+ outage), you can get the entire fee waived.

Microsoft gives a 99.9% monthly uptime guarantee, which means maximum allowed downtime is 7hrs.

I’m extremely curious what actual veterans could even come up with to explain the situation as it currently appears.

I’ve seen the chats about cascading failures and it’s impossible to not expect AI is involved somehow (likely brain drain and what not), but what are the options?

It sure looks like “fired everyone who knew what they were doing” but I feel like at this point you’re breaking open the check book and calling them, and even if they did that I’m assuming this shouldn’t be a “normal” event?

Looking forward to the postmortem on this one and how much AI contributed to the problem and troubleshooting.

I've been saying AI hasn't been tested properly. All the showcases I see aren't really pushing it far enough and they're always things you could in theory do by searching answers online. The real value is in all the problems that cannot be solved by online search.

This is one for the history books. The status updates seem like AI nonsense, it doesn’t seem like they’ve even root-caused what’s driving this.

I know this is only tenuously related, but after a week of them making changes (starting with DKIM) and blocking third party mailservers (and this week I got my first "your IP is blocked" but when you use their request to remove they say it isn't), I hope it goes down long enough to make more people migrate away

This was the event I needed to design and implement my email DR plan, by selfhosting Postfix again as a backup MX server.

When you outsource all your coding, architecture and thinking to AI, you inevitably stop understanding what's actually going on.

That might then become a problem when things stop working.

downdetector says everything is recovering

but I still can't send an email in Outlook

the hell is going on?

Really hoping it's a monumental fuckup that lasts through the weekend well into the next week.

The corrupted executives need to learn their lesson on consequences of their actions. Laying off the core of your system?. Say goodbye to triple-9 SLA.

I have zero sympathy for big tech

[flagged]

This is what happens when you outsource your business to India and lay off your seniors.

https://status.cloud.microsoft/m365/referrer=serviceStatusRe...

> Engineering teams are cycling affected infrastructure through traffic drain, restart, remediation, and staged return-to-service while monitoring closely.

so they have no clue what is going on...

I heard a story, which you can't attribute to me of course, I just heard it -- we had a thing (literally yesterday) where a person from M$ came selling us some AI snake oil (we are very much an azure pilled org, but not super AI pilled).

The meeting was obviously in teems, because they sold us on that already and most of the presentation the screen sharing didn't work for half the people, which generated a continued stream of office-grade jokes for the whole duration of the presentation (30 minutes or so).

As a guest the presenter was totally oblivious to this fact, as they didn't get access to the chat messages, because the policies are in place (we are a serious company).

Hear that - that is the sound of every Engineer at Microsoft right now typing "We have a global incident - fix it" furiously into their chatbots!

I guess we better get used to this new normal and pray your travel day or important appointment is not going to be on an outage day, whether that's MS, Crowdstrike or else.

Just use AI, duh?!

Call Sam Altman for some RAM first!

Interestingly, github is NOT down.

Maybe they morged[1] a bad commit somewhere

[1]: https://nvie.com/posts/15-years-later/

Vibecoding is going well it seems

You don't say.

Get Copilot to fix it.

hey Claude fix my production issue in production make no mistakes

investigating..

goofing..

finagling..

hey claude what the hell you deleted all my configs permanently!!

metastasizing.. You’re absolutely right. I went above and beyond what you had asked and started completely fresh. Unfortunately I didn’t take any backups and deleted all existing distributed configurations. Oopsie!

Gmail down for me also. China must be mad about the Nvidia dude.