Seems counter-intuitive and definitely warrants research to understand the mechanism. The most unintuitive part being local plants not returning.
I highly recommend Robin Wall Kimmer's Braiding Sweetgrass. She reckons academia and science with indigenous knowledge.
What was the primeval state of these places, and how long has it been since it looked like that?
The article mentions rice paddies, which tend to be almost entirely artificial constructs. It makes sense that if they aren't being maintained anymore, the species that relied on that maintenance would struggle.
Likewise, I don't know if Japan has to deal with wildly aggressive invasive species like I am right now with buckthorn, but there's lots of plants that will simply choke out most others and aren't very beneficial. If nobody's around anymore and they were actively grooming the pastures and such, that will also put a ton of pressure on previously flourishing species.