Although the trope is hilarious I think most people just don't bother since it doesn't matter to them. I never had a problem setting the time on my VCR and using it to automatically record shows while I was at work.
Yes it was no more difficult than setting any other digital clock. Even today, my microwave, kitchen radio, and several other devices all read "12:00" because I just don't bother to reset them every time there is a power glitch.
I remember having trouble with mine, often mixing up the various hours (clock time, start time, end time, recording duration). Yes it was not rocket science, but it was used not enough to remember how to do it, and the manual was never ad hand when needed.