Years ago people warned that we had to limit the population to save the planet.
Now we're naturally reducing population growth, and people warn of disaster.
Gradually decreasing populations means that overall, fewer workers have to support more non-workers. We've had decades of productivity growth, with more to come if we're not stupid, making that extremely feasible.
But that productivity goes more and more to capital instead of labor, making it much less feasible.
The key word is gradual. Population decline in Japan may be faster than is economically manageable, depending on whose projections you read.