If the limit of someone's behavior winds up making everyone happier-off, I don't understand why I ought to care. In that sense, calling it "manipulative" seems either inappropriate or not very useful.

At least with something like adultery, there's a pretty obvious ill consequence of someone finding out what's going on behind the scenes. But if I looked behind the curtains of someone like OP and found out that the reason they're so charming is because they thought about people a bunch: I couldn't be burdened to care.