The Rust compiler, while not extremely fast, is not extremely slow either. As an example, my toy compiler is 130K LoC of Rust and compiles in under 7 seconds on a cold build on my M5 Max. An incremental build is nearly instant. So then why is it that so many projects have a minute compile time with far fewer LoC?
The answer: dependencies (My compiler has virtually none, intentionally)
Rust has a very small stdlib, so you are forced to bring in dependency crates, and not just a few. Many programs have 50-100 or more, and they all have dependencies, and so on. Contrast with Go where it has a culture of few dependencies, and those project have few dependencies. Your Rust compiler is doing a lot more work for what appears to be the same program as in other langs, but ultimately is a lot more code (typically).
To make matters worse, cargo does a decent job of parallelizing the builds, but it can only do so a) at the crate level b) only as much as the dependency graph allows.