logoalt Hacker News

ncr100 • yesterday at 2:29 PM • 5 replies • view on HN

They talk about their relatively low-end testing environment PASSing the July code, and being different from their high-end production environment machines, sadly hiding the concurrency bug during the test phase.

Is this commonly acceptable? I'd imagine it's a costly trade-off?


Replies

bayindirh • yesterday at 2:53 PM

Depends on what low/high end are. If low end is 4 core and deployment is 128 cores, this is not acceptable.

But if low is 64 and high is a whole cluster, now that's a different kind of maths.

Edit: The article states the following:

> "lock contention inside an internal Windows process which caused the machine's resources (CPU & memory) to be exhausted."

So, it's more the former. The testing machine is probably a 8 year old server on an office shelf, and prod machines are 64-128 core cutting edge Xeons/EPYCs.

Lock contention. It was MUTEX spins in Windows 95, it's lock contention in 2026. Why this is still a thing, Microsoft?

(Man, I felt old while writing this.)

➕ show 3 replies
notyourwork • yesterday at 4:02 PM

Testing at production scale is extremely difficult. It’s also financially expensive. In my experience, you balance the two where big changes are tested with some simulation setup to gain confidence. Though, the ci/cd pipeline never has close to full scale automated testing.

➕ show 2 replies
munchbunny • yesterday at 10:19 PM

At Microsoft's scale, it's extremely difficult to simulate production scale because the sheer size of the production system is overwhelming, and many issues emerge from the combination of sheer volume and the complexity that reality tends to introduce.

Microsoft tests its stuff on their own employees (old citation, but still true today: https://devblogs.microsoft.com/oldnewthing/20110802-00/?p=10...), and that includes Exchange online. However, even though Microsoft is a pretty big company, it's still teeny compared to its customer base.

awesome_dude • yesterday at 11:18 PM

I read that part of their statement as "QA were set a priority to ensure existing hardware wasn't affected, which reduced their (QAs) capacity to test on new hardware which was assumed to be safe"

They find QA dirty in their statement, because they don't state who set that priority, nor why the new hardware was felt to be less likely to be affected (what did their risk matrix look like for their hardware profiles i wonder)

flycensed • yesterday at 9:55 PM

[flagged]