logoalt Hacker News

lkjdsklf • yesterday at 9:43 PM • 1 reply • view on HN

Considering the bug was latent for several months in production before they ran into it, unlikely they would have caught it in a contrived test


Replies

deeringc • yesterday at 10:17 PM

Seems like the issue happens readily at high load on a system with many cores. Once the conditions were right it seemed to impact a large proportion of their fleet. They just didnt have that combination in their test/validation pipeline. They likely have some sort of load test in their pipeline, but it's running on machine that's much smaller than their production machines... so it didn't trigger the lock contention bug.

The opposite is true running in production for the last several months. They were runnign on big production machines with enough cores, but never quite had enough load to trigger the lock contention. But once both conditions were true, big machines and high load, due to some unrelated failover elsewhere suddently bringing more load, we then had the magic combination.

It's really not unreasonable to expect an enormous hyperscaler like Microsoft to do load testing on hardware that exactly matches their production systems.