I think the reason people get angry about it is that it feels like an asymmetric ask. It feels a bit like saying "everyone needs to come I to the office because I like there to be people in the office, regardless of whether that's what they want." (not saying that's what you're saying here, just trying to shine some light on how it may be perceived)