It's sort of a silly opinion really. The British Empire spanned the globe. It's akin to being shocked to see blacks in the US.

Because a lot of people who are not white are born in Britain, making them native Brits?

that's not the controversial part, it's the distinct overtones of "go back to India/China/etc" that comes with it.