A) people had an issue with immigration when it was mostly from the EU as well, one of the main drivers of Brexit

B) leaving the EU did not mean that non-EU migration had to go up

C) immigration going down for one or two years doesn’t change the fact that millions of people are in the UK that most British people didn’t want to enter in the first place over a period of 15+ years

D) net immigration going down still means lots of people are entering the UK from elsewhere, while people from the same culture (ie grew up in the UK) are leaving which still changes the culture of the country

Note: I’m not saying I personally agree with any of those points but it’s clear what the issues are if you’re prepared to listen. There is opinion polling on all of this.