logoalt Hacker News

jiggawatts • today at 10:02 AM • 0 replies • view on HN

I can think of several scenarios that could explain the outage:

A bad antivirus pattern file can cause enormous CPU spikes. E.g.: a very inefficient regular expression with too much backtracking. This tends to get pushed out every few hours to every server, irrespective of role. Hence, "rolling back" the servers to a "known good image" doesn't help, because within a few minutes they get the pattern update anyway and die with CPU overload. (IIRC this type of issue caused a Cloudflare global outage recently, and is similar to what happened with Crowdstrike.)

Similarly, someone here mentioned a very recent OpenSSL security update that also caused enormously high CPU load. These critical patches have been rolled out very quickly with minimal testing because without it, systems are very vulnerable to nation-states utilising AIs to rapidly exploit new vulnerabilities. Microsoft (probably) doesn't use OpenSSL much, but if they do, it would be everywhere because its use in layer-7 load balancers tends to be "pervasive" and hence a sweeping update can absolutely trash performance of everything in the data centre. (However, this category of issue would have been fixed by rolling back the server images)

It could also have been a simple underlying initial cause, but then the "backlog" of emails that accumulated throghout the day would overwhelm systems that come back online, especially if they can't start them all at once... which they've said they're not doing. So, you start 10% of your capacity, which tries to process 10x the normal email volume => splat.

Most other explanations wouldn't account for the global outage.