Barring physical limitations, what you can and can't do is ultimately determined by what the society you are by and large a part of deems to be acceptable behaviour.

Government rules and social norms can change over time, it ultimately doesn't matter what you feel is "right" or what some law says is "right", it's really about what you can get away with.

A large part of what you can get away with is determined about whether or not you will ultimately be penalized for your actions (possibly through violence), and laws can keep people aligned on what is or isn't going to be accepted and when people deemed to be acting in a socially unacceptable way are likely to be penalized in some form.

While "rights" may be somewhat philosophical, they can have very real physical "weight" behind them in the form of other people "enforcing" them.

And finally, in case you are mistakenly under the impression that I think it's okay for anyone to do anything they want so long as they can get away with it, I don't, but that discussion drifts into the territory of morality and ethics which, while related, are nevertheless different and very large topics of discussion in themselves.