logoalt Hacker News

hiddencosttoday at 4:56 AM1 replyview on HN

> A stochastic search process with an executable optimization objective over space of programs S can only maintain or improve the objective

Reasons this doesn't follow: (1) Benchmarks never match real world use, and many optimizations the improve benchmarks degrade cases that aren't measured (think about how CPU cache behavior can be surprising) (2) In software performance optimization, frequently there is significant noise, from many sources. This makes it difficult to guarantee that a measured change is actually an improvement.


Replies

fc417fc802today at 5:47 AM

Both of those things are indicators of deficiencies in the testing process.