I don't even explain why I'm taking a day off or a half day or whatever.

It's no one else's business to decide if they agree the reason is valid. Don't volunteer stuff like that especially when it would be something they'd likely agree with. Don't validate their right to decide or even be aware of anything like that. Next time maybe your need might not be so convinvcing to anyone else, but you've already established they have a right to know and a right to veto, so then trying to be private stands out as different from other times.

It doesn't mean being inflexible or uncooperative either. If there is actually some problem, they can negotiate and try to make the case and convince you that they really need you just that much just that day, and you can voluntarily decide to change your plans, but without ever volunteering what those were. And make it cost. Keep score. Claim a return favor.