> RISC-V is pretty unoptimisable though

Is it? So there's no scope for different levels of pipelining, out-of-order execution with various numbers of execution units, branch prediction of various levels of sophistication, L1 cache of various amounts, multiplication and division implemented in various ways from slow but low chip area and power consumption to fast with high chip area or power consumption? Because those are all things that have major performance impact even on plain "RISC-style" x86-64 code.