I'm right there with you, and it's why I go to great pains to articulate the entirety of my position on immigration when I get into these sorts of debates. The simpler someone's position on immigration is, the less they understand it at length or the more extremist their viewpoints tend to be.
It's wickedly complicated, isn't it? I'm distressed by anybody who doesn't change their position from time to time.