If supersonic flight is so expensive that less than 1% of the population can enjoy it its not clear why the rest of us ought to tolerate it
You have to start somewhere before you can build towards an economy of scale that makes it affordable. If people used your same logic about regular airplanes, we wouldn't be using them today.
Get those rich people out of the private jets and into the supersonic ones. Then the rich people currently in first class can move into the private jets.
Eventually, that means that there will be some seats in the back of the regular plans for everyday wage slaves.