Two counterpoints: Searching for "unsafe" is a lot easier when you want to audit the code (there is even a lint you can enable to forbid all unsafe in a crate (library)). And there is a pervasive culture to avoid unsafe where possible and document all the unsafe you do have explaining why it is in fact ok.

There are also experiments in formalizing the safety comments with attributes. To me the current prototypes look halfway towards formal verification, with unsafe functions specifying named requirements that must be upheld when calling them and the callsites needing to "discharge" them by name. Time will tell if this is a good idea for general code and it it catches on.