Premature optimization is that occasional dessert we serve ourselves because it tastes so good.

That said, I agree with you that the maxim is often used to justify poor planning or absence of planning. Premature pessimization is bad too.

We don't have the bandwidth to test every micro-decision. That's what the learning of domain specific principles are for. Then some choices you reason through, some you test and when confronted with a perf problem, test those reasoning based choices that benchmarks point fingers at.