Everyone is against "mass" immigration as in "let's go to country X and gather a million people and bring them to country Y", that's not controversial
It is controversial when you start pointing to individual immigrants already in country Y and saying "hey you, get the fuck out".
Which is of course why people like Musk always try to frame things as the first one. They always say things like "the president is importing one million illegals" and never say things like "one million different people decided to come here, all for different reasons"