This is all, frankly, nonsense.

If you're using userspace software to decrypt the keys into memory then it's all just security theatre. Any sufficiently motivated actor, ot a comprised package in the software chain, can capture your private key in the clear. Especially if using ssh-agent that keeps it resident in memory 'for convenience'.

Proper HSM integration with SSH requires challenges signed by the HSM without the private key ever being emitted.

Just keep your SSH keys on disk, one per remote host, with a different encryption password for each target host. If you're worried about threats that can defeat that then they'll just come to your door and beat the passwords out of you.