Seems like maybe crypto having a single point of failure (keys) is a problem rather than hardware.

You will never be completely secure against failure, the only way to protect yourself is to be able to robustly recover from failure.

I'd rather be able to say "every single government is in my hardware and they still can't touch it."

Perhaps possible with multisig, but now you're basically back to trusting banks.