For Japan, shrinking population will mean an aging population with no one to care for them.
A people can recover from that, though. That's not an existential threat. They may be in for hard times, but they will survive.
Can the same be said of all the populations importing millions of foreigners to prop up "muh GDP"?
A few generations ago they were enslaving people [0]. This is what racism can lead to. Maintaining their heritage, their way of life, some would say.
Hopefully their population decline doesn't hit them too hard. But that, and their lack natural of resources, could lead to bad things. Fascists and their ilk thrive in such situations.
[0] https://en.wikipedia.org/wiki/Slavery_in_Japan#World_War_II
Not if we have robots and AI.