You have to start somewhere before you can build towards an economy of scale that makes it affordable. If people used your same logic about regular airplanes, we wouldn't be using them today.
I'm still waiting for Tesla's plan to come true where the rich paid for development before scaling up brings the prices down. The luxury economy of scale is about as realistic trickle down economy
Supersonic flight will never be suitable for the masses compared to normal flight due to basic physics. It will always require much more expensive planes and more fuel.
It's not more viable now wealth inequality just means that there are more rich people to benefit.