The concern usually only comes when you fall on the "outside" of the prevailing opinion within the group of people voting on comments.

edit: my response isn't about lobste.rs. I don't have an account there.