Active Benchmarking

> configure the benchmark to run for a long duration in a steady state: e.g., hours.

Benchmarking as a developer activity isn't going to cut it long term, that is at best a proposal to change.

If you do an active benchmark, which is basically running a workload when the SuT is tied up to a "data polygraph" of sorts, then you tend to accidentally build production observability tooling in the process.

The best kind of "benchmark" comes from a fleet in action, rather than something which was run in isolation. But those tend to be entirely a 1 of 1 test scenario rather than a real replicable result.

Most of my past performance work revolves around caches, those are exceedingly sensitive to workload patterns and what worked in May will fail in October.

With the whole cloud thing, the "workload pattern" might be someone else's.

My team put together a blog post last week which concluded that our coding models work better in PST afternoon rather than the mornings.

I haven't pursued this as a conscious policy but I can confirm that every time I've done it I've always discovered something that would have invalidated the benchmark if I wasn't looking at it while it ran. Article mentions it but I've seen something as simple as "gee that's strange that should be using all my cores not just one" several times over, or equally stupid things.