This is sort of the exact opposite of reality: the point of safe Rust is that it's safe so long as Rust's invariants are preserved, which all other safe Rust preserves by construction. So you only need to audit unsafe Rust code to ensure the safety of a Rust codebase.
(The nuance being that sometimes there's a lot of unsafe Rust, because some domains - like kernel programming - necessitate it. But this is still a better state of affairs than having no code be correct by construction, which is the reality with C.)
It's just moving the goalposts. "If it compiles it works" to "it eliminates all memory bugs" to "well, it's safer than c...".
If Rust doesn't live up to its lofty promises, then it changes the cost-benefit analysis. You might give up almost anything to eliminate all bugs, a lot to eliminate all memory bugs, but what would you give up to eliminate some bugs?
Which domain doesn't necessitate unsafe? Any large Rust project I check has tons of unsafe in its dependency tree.