> They increase prices for foreign goods
Not only for foreign goods. If there's a tax on imported X, domestic X producers can now charge more because the foreign producers must raise their prices to pay for the tax.
Another way to look at it is that any tariff reduces foreign competition, with all the knock-on effects that implies.
It's not the foreign producer who must raise her prices to pay for the tax. It's the local importer. The net effect on the consumer is that the price of imported goods goes up so then, as you say, the local producers can raise their prices and increase their profits.