Anecdotally speaking, I've personally lived in multiple countries on multi-year-long temporary resident visas without the right to work, because getting the right to work is a separate process from being allowed to be somewhere. So have family members. And in case it makes a difference, which it might, one of those countries was the US.

In fact, did you know, in Canada it doesn't count as work for visa purposes if you work over the internet for a foreign company and get paid into a foreign bank account and don't compete professionally with local interests? (This is a free hot tip for anyone wanting to live in Canada. It might be true in some other places too, but I don't know.)

Spouses of folks on many kinds of visa are allowed to be here, but aren't allowed to work.

It's borders that aren't supposed to be there, not people.