The endorsement system already works along that line: https://info.arxiv.org/help/endorsement.html

It's probably not perfect but in practice, it seems to have been enough to get rid of the worst crackpotty spam.