Yes, it's a meaningful distinction. No you are not into "anything happening" in practice.
Your compiler emitting a load operation and it failing isn't "anything". The failure being handled by code that the compiler authors can't predict doesn't make it "anything".
And if you lose optimization opportunities because of this it's because your optimization is broken. By the way, if you lose optimization opportunities because of this, that means both codes are meaningfully different and you knew it all the time.
I mean... You can turn a one byte out of bounds write into code execution.
https://daniel.haxx.se/blog/2016/10/14/a-single-byte-write-o...
And if you get code execution, then you by definition have "anything".
Compilers elide loads all the time this is one of the more basic optimizations a compiler can do. We just mostly think those are "good" optimizations.