Generally the pollution comes together with other indesirable effects. Stench, noise, etc.

The rich don't need to understand that roads or ships generate deadly air pollution. They don't like living next to a highway or a container terminal, full stop. They do however love living next to a park or a lake.

In fact, so do poor people. But they can't afford it.