I'd argue that civilization is the story of people finding ways to overcome what they previously had to tolerate.

I disagree. The more that I travel, the more that I notice the trend: The richer you become, the quieter you become. Even in the same country, income level seems to strongly correlate with how loud you are in public (and private).

It's worth noting that the initial supersonic bans came as a result of sustained sonic boom testing over the same population: https://en.wikipedia.org/wiki/Oklahoma_City_sonic_boom_tests

If supersonic flight is so expensive that less than 1% of the population can enjoy it its not clear why the rest of us ought to tolerate it