This is only a harm if you are ambitious and career-oriented. I'm remote and know it won't be conducive to promotion, but I also get to:

1) live in the low CoL area that I grew up in,

2) be near family and friends (and therefore free, high-quality childcare),

3) avoid a hellish commute in one of the sprawl-y hellscapes that grow up around tech hub cities,

4) live in a paid-off house instead of 50%+ of my income going directly to rent or a mortgage, and

5) have a massive nest egg due to all the money I'm saving.

Could I get faster promotions by going back to the office? Maybe, though I see the careers of my at-the-office colleagues around me stagnating just about as much as mine. But...I don't want to be management. I don't even necessarily care for promotion as an IC unless that's the only way to tread water with inflation.

The only major downside I think about is that it will obviously be harder to get a remote position if I lose the one I have, but we're financially prepared for that. With a paid-off house in a single-car neighborhood, we can make ends meet with a normal job stocking groceries or something. At the worst case, I have connections to get a job at the factory a town over, though that would mean getting a second car.

In other words: I, as a worker, do not care about maximizing the value of my human capital stock. I am not cattle. I am not a slave. I have preferences that are unrelated to my ability to receive praise and promotions from my boss. In short, I deserve respect from my employer, whether they are currently being forced to give it or not.