That's not how it works. A larger codebase to scrutinize means that there's more chance of missing a memory safety bug. If you can keep the Rust unsafe block bug-free, you don't need to worry about them anymore in safe Rust. They're talking about attention getting divided all over the code where this distinction is not there (like C code). They always have been.
On top of that, there is something else they say. You have to uphold the invariants inside the unsafe blocks. Rust for Linux documents these invariants as well. The invariant was wrong in this case. The reason I mention this is because this practice has forced even C developers to rethink and improve their code.
Rust specifies very clearly what sort of error it eliminates and where it does that. It reduces the surface area of memory safety bugs to unsafe blocks, and gives you clear guidelines on what you need to ensure manually within the unsafe block to avoid any memory safety bugs. And even when you make a human error in that task, Rust makes it easy to identify them.
There are clear advantages here in terms of the effort required to prevent memory safety bugs, and in making your responsibilities explicit. This has been their claim consistently. Yet, I find that these have to be repeated in every discussion about Rust. It feels like some critics don't care about these arguments at all.