Xenophobia is a tool of demagogues, not a rational policy.
So while it's an issue, it's not even in the top ten of of the UK's most important practical issues.
And the racism it generates - including Brexit, and currently the collapse of income from foreign students, which was a huge earner for the university sector - does huge and lasting harm.
It's a political shiny penny used for distraction, not a realistic solution to anything. And the people who make careers out of promoting it are provably cynical, corrupt, incompetent, and disinterested in solving real problems.
The problem with this argument is that for essentially my entire lifetime, the UK population has wanted less migration. And what they have got instead is gross migration running at 1 million per year. If they are just fundamentally wrong on this and have been for 50 years, it seems like democracy is a bad idea. Alternatively, if you feel you have to accept democracy, then maybe it would be good to give people what they consistently say they want on this issue.