I think it's mainly that JS has a terrible developer experience when it comes to parallelism and shared data, which these other languages don't need to worry about. Most build tools are pretty amenable to being parallelized but with JS you end up either being constrained to process-level parallelism (e.g split the filesystem into chunks and just run the program multiple times) or you have a bunch of inefficient message passing between processes. Even worker threads are unable to share memory except through the weird and awkward-to-use SharedArrayBuffer and Atomics APIs.

On top of this, a lot of JS tools have crazy plugin systems that let you hook into performance-sensitive places and run arbitrary code. This is slow and also makes parallelism tricky. For example, babel plugins can make arbitrary changes to the AST. Ordering them is already kinda tricky; many plugins just give up and run their entire logic as a callback against the root AST node. The fact that swc, esbuild, etc. come batteries-included with transforms for module syntax, Typescript, etc. actually helps with performance; you can leverage assumptions about these fixed sets of transforms to run them all at once and avoid multiple passes over the entire AST.

So rewriting in a language with better ergonomics and being able to ditch some excessive flexibility is what gets you the massive speedup, it's not just magical Rust dust