Yeah. It's called brain drain. Talent has options. It weighs pros and cons. When the relative attraction of a country and thus institutions within it drops, they choose to go there less.

To be clear, I would still choose to do my PhD in the US. But this is a marginal effect, people weigh many factors. If you think, for example, you're going to be constantly worried about visa issues, you may just choose Europe or China over the US.

Edit- sorry NZ and australia, forgot about you