This is a tired and unhelpful refrain. Only rich people fill their cars with gasoline without wincing at the price. Only rich people get to own 7 houses. Only rich people get to fill their pools in the middle of a drought.
There are a lot of things that “only rich people get to do”. Reducing the number of people who engage in destructive activities is a good thing, even if it means only rich people can still do it.