Give your user NOPASSWD if it's really that bothersome. You can also potentially set it up to use a fingerprint reader if you have that hardware.