street is public. Nothing is "done to you"

Freedom to walk anywhere means someone can walk onto your property ("done to you") You can curtail that freedom, because you are essentially giving up ("inability to do something with stuff someone else owns") some freedom to get some other freedom ("ability to own stuff that will not be used by strangers").

It's a tradeoff. A good one. Tradeoff of say "nobody's anything is private now because that allows govt a slightly easier time to catch criminals" is not a good tradeoff.