Yubikeys (or something like keys in the Secure Enclave of a Mac or iPhone) are themselves not accessible by software, you can only ask the hardware to perform operations with them (e.g. decrypt or sign something).

This means someone with access to your filesystem can’t just copy the key off.

Of course a passphrase is another protective layer. But passphrases can be keylogged, or the decrypted key (or passphrase, depending) in the memory of an ssh agent can be can potentially be read out by an attacker.

With separate hardware, the assumption is that the hardware is much much harder to attack, so the keys stay on there and only on there.

You can require physical user interaction to use the key. If the mossad is after you, this doesn't really help you because they'll just replace the ssh binary and intercept one of your requests, but against a less motivated attacker, it makes it harder to pivot from one machine into others.