That's not actually sufficient in the general case where the pointer may not be the type of the underlying object. You also have to respect strict aliasing even if the bounds are correct. This isn't true in the same way in Rust because memory is untyped. You only need to ensure basic memory validity (range, initialization, alignment, etc).
Rust has pointer provenance which implies very similar constraints to the "typed memory" wording of C/C++.
Yes, you also do not want to do random casts, but this is even easier. I do not get your point out memory validity in Rust. What if you write where a pointer is stored, or even a boolean?