I agree that that's not a best practice. It's not what the downvote mechanism was intended for.
Flagging is used by people who have no rebuttal but are mad.
That's why I have only flagged one or two posts, ever, but not because I was mad, but because the comment was just plain beyond the pale.
And my posts against portaying violent rape in film got flagged.
Make it make sense, because I understand the failure of this system because systems are my trade-in-craft.
In a way, iirc, it really is. It's as much a "I disagree" as it is "I don't like this". That said, I would like to see more people actually respond in addition to a downvote.
I don't think that's generally a function of the moderators though.