I think it's time to update your expectations. Hardware security modules have been around for quite a while. You shouldn't assume that your chosen password is the only input to the key derivation, or even that the raw key itself will always be obtainable.