Anyone who has ever mixed music seriously knows the effect of psychoacoustics - your brain is such a proficient signal to noise optimiser, that differences in audio quality are perceptible only for the first few minutes of hearing a given sound source.

This is why mix engineers have multiple sets of speakers, so they can periodically reset their perception. Additionally, it is well documented that the top scoring speakers on Audio Science Review (where people who think linear reproduction of audio is the only job of a speaker) do not align well with the top tool choices for audio professionals, and this is not because they are governed by supersition.

There is not currently a metric for many of the things that become essential as soon as you start listening critically and having to make binding decisions from your perception - will it fatigue me, is the speaker lying to me, dimensionality, etc. Saying these are problems with your listening environment is punting on the problem of the actual duty a speaker is required to fulfil.

I don't see why an audiophile, who is a critical listener, should be any different. They're obviously doing it without a generated artefact from their listening, so there's no data based feedback loop for the quality of their experience, but it's a hobby, so everyone's preferences have value.