I think it's an understandable prior. Historically, "low level stuff" was near-exclusively (see my comment below about OCaml...) written in unsafe languages. Even if that wasn't always literally required, it sometimes was, and so thinking this is the case was a reasonable thing to think.

It is only relatively recently that we have gained more realistic options in these spaces, and so not fully understanding the implications, or preferring the historically normal choices, is understandable.

I'll play devil's advocate. I think emitting machine code intended to run is unsafe because you could emit unsafe machine code, which could run. It's the whole system that is either safe or not, not the individual components. If your system gets hacked by a buffer overflow in the end, nobody cares whether it was the linker that overflowed or the code emitted by the linker.