What you describe is a different attack surface than the much more problematic on that password managers solve well: password re-use across websites where the blast radius is super high.

If I had to trade off the risk of a password manager being hacked versus password re-use, I'm going to strongly recommend to people generally to use a password manager. Yeah there's a lots of "if this, if that"--but that's almost a universally better risk for most people.