They can be detrimental too, especially if they're linked to beneficial traits. The test is ultimately whether or not the harm done is sufficiently disadvantageous that it interferes with reproductive fitness. Baldness is arguably detrimental, but it's linked to a bunch of recessive genes that function in other ways, and it doesn't impact us until we're likely to have already reproduced.

That's a simplification, but you get the idea.