Honestly looking over time I think that phrase did more bad than good.

Yes, of course you shouldn't optimize before you get your critical path stable and benchmark which parts take too much.

But many, many times it is used as excuse to delay optimisation so far that it is now hard to do because it would require to rewriting parts that "work just fine", or it is skipped because the slowness is just at tolerable level.

I have a feeling just spending 10-20% more time on a piece of code to give it a glance whether it couldn't be more optimal would pay for itself very quickly compared to bigger rewrite months after code was written.