I don't think I'd have the wherewithal to jump in and do something if I were a bystander. I'm not the sort to throw hands, I don't carry, and these disruptive types are already a bit feral.
I'm not sure it's contempt they're expressing, or if they're expressing anything at all. There really are people who enjoy and defend it, too; "it's just a guy playing music, mind your own business." Truly alien.
I've found that looking the person in the eye and giving a quick "hey, forget your headphones?" sometimes does the trick, and has yet to start a fight. Everyone has to act in ways they are comfortable with - but mass inaction is what creates space for this shitty behavior in the first place.
My business includes my ears. If you don't want me in your business, keep your business to your ears.