That's not a good argument. First, music is not speech. It doesn't require linguistic processing. Second, you can hear chords instead of individual voices, so polyphony can be perceived as shifting harmony, and it usually is.