There is a global market for energy, a country X that would have bought from Iran but couldn't because of refinery damage has to buy it from someone else, pushing up the price at the margin for everyone.

While they have been under heavy sanctions, they have largely gotten around this, for example by trading with China (who have just said they will not abide by our "unilateral sanctions"). So their oil has still been going to service global needs, and its lack will still have an effect (as will the missing Russian oil products).

Oddly, since the Iranians (and Russians) have been selling their oil products at a discount, they were actually having the effect of holding down the price of petroleum prior to the war. No idea about how much in real terms, but...

Iran was still selling oil to China and others. Those now have to buy somewhere else, so it does affect our prices.

That's the main issue, yes. But where was all that oil Iran was producing going before their infrastructure was destroyed? Are you claiming they only ever used it domestically and didn't export any of it? That seems unlikely.

Iran still exported, and the nations that bought will now buy where we buy as well.