I've found that looking the person in the eye and giving a quick "hey, forget your headphones?" sometimes does the trick, and has yet to start a fight. Everyone has to act in ways they are comfortable with - but mass inaction is what creates space for this shitty behavior in the first place.