The case for the local guy is not calculated correctly for EU VAT, leading to the misunderstanding. The cost of the input increases proportionate to VAT, with the VAT refund cancelling this out exactly.
Let's assume as you did that the actual input has the same real cost to manufacture (ignoring e.g. local labor cost differences, fuel cost differences, government incentives, etc.), and that your example therefore needs exactly 80 USD worth of real, untaxed input in both cases.
The US producer buys this for 80 USD out of pocket as there's no VAT to pay, adds 20 USD profit and sells it for 100 USD. An importer adds 20% VAT, making it 120 USD.
The local producer has to pay VAT so their price for the same thing is 96 USD out of pocket. They get 16 USD VAT back from the government next time they file VAT (irrespective of what they sell), i.e. the "relief" undoes the VAT entirely. They add 20 USD profit making the price 100 USD, add 20% VAT, and what do you know, same 120 USD price tag.
In both cases the final consumer price is 120 USD, and after VAT is cleared the input seller gets 80 USD, the producer itself gets 20 USD profit and the government at the point of consumer sale gets 20 USD VAT.
If the local guy exports the goods to the US the VAT is refunded, making both prices 100 USD. Input seller earns the same, profits remain 20 USD, but no VAT is earned.
This is also why all companies only discuss prices excluding VAT, as the VAT is purely symbolical if it's not a consumer sale.