I don't just imagine it, I do it, by using gpg-agent as my ssh-agent and using the private key generated by a Yubikey. Another way is to use tpm2-tools so only your laptop running your own signed boot chain can use the key. It is desirable to lock private key material in a physical thing that is hard to steal.

You can choose not to do this, and that's fine. Hardware attestation is dead because Apple refuses to implement it, so no one can force you to.