A billionaire buying up housing can do a lot more damage than a persons drifting through town