This is the only counter argument, and in fact the consensus is that they are the fairest (they tax consumption rather than production, we tend to tax producing wealth more than spending it) if you offer rebates or lower them on essentials (bread, milk, eggs, healthcare).

They also have the effect that everybody has to pay them, including tax evaders, tax loophole abusers, criminals with undeclared incomes, etc, everybody has to pay it.

But yes, without offering sales tax rebates or with taxing essentials then your argument is true and they become less fair.

Albeit, the elephant in the room is always the definition of fairness itself.