Really depends on what you benchmark and how reliable you want the measurement and what domain you are benchmarking. For example, criterion can sometimes spend quite a bit of time because it needs to stabilize the measurements. The author’s claim is “well you don’t need that accuracy” but I’ve seen wasted time chasing ghosts or making claims on performance improvements that were either neutral or net negative due to this noise.

> Anything faster than, say, 10ms risks being skewed by fixed costs (e.g, interpreter startup).

Sounds like the author’s experience is strictly in Python. For example with Java you have to make sure the JIT has sufficient optimized your program.

Additionally there’s plenty of situations where it can take a really long time to generate a representative dataset worth benchmarking and it can take time to evaluate the performance (eg databases). Short and quick microbenchmarks can be useful as building points, but at some point you need to evaluate steady state performance of the full thing. Other domains this comes up with is game rendering performance where a 300ms sample tells you nothing about whether you have frame drops after minute 25 or have a memory leak.