I think it’s more of a time vs code tradeoff, if done properly.
For example in the Vec case, you could theoretically build an alternative which encodes the “three sections” property internally, and ensures correctness at construction time for the pointers. Not as completely safe as a Vec, but you can still get similar benefits for the “business logic”.
But I agree, just having a custom structure that does not provide a safe wrapper around this would be sacrificing standard guarantees.