I had tried this but found it a little bit weird - switching back and forth on the same device between the 'hard drive w/ full files' library and the 'primary drive optimize storage' didn't really seem easy.

IIRC Photos.app will not even open if the default library you are pointing at is not there (i.e drive was unplugged). Are you able to just open up the library file directly and it will work as expected?

I also recall when changing Photos.app back to the HDD Library it did a ~2h 'rebuild' session before it even started downloading the new photos, but maybe thats acceptable with the 'every so often' approach.