If there is no agreement, then the magic term is "tacit collusion."
Why not ask the same ridiculous amount of money your competitors do? People seem to be paying for it. Their fault. If suppliers have sufficiently different products, they can make some more expensive, others cheaper; on average, everybody pays more. A high barrier to entry might help such practices.
That doesn't mean I'm saying this is what is happening. Sometimes things just suck, and somebody bought the world's supply of RAM wafers to use as frisbees.
We have to decide what part is damaging to society: the actual physical agreement, or the effects of the agreement?
If it's the actual physical agreement that's the problem - the system is working as intended.
But if we are looking to prevent the negative outcomes associated with price fixing and collusion, our system is failing us.
They are never going to find proof of conspiracy. The people involved covered their tracks, and doing so is trivial. So the best we can do is punish the appearance of collusion. And if the goal is to actually prevent harm to customers, that's a better solution anyway, since it encourages leaders of companies to behave in a manner that's the opposite of collusion.