Approximately two billion people who speak tonal languages demonstrate that it's really not, I think, given that e.g. native cantonese speakers reproduce to within a quarter semitone across a lifetime barring something like neurological or significant hearing issues. They don't all have perfect pitch, that seems to be related more to music training early in life, but something like 60% of Mandarin speakers who were trained musically before age 6 have perfect pitch. In those cultures, not having at least relative pitch is a learning disability similar (or perhaps even more problematic) than dyslexia is in e.g. english speaking populations.