Sort of, at least some degree of relativism exists though how much is debated. Would you ever talk about sea having the same color as wine? But that's exactly what Homer called it.
https://en.wikipedia.org/wiki/Wine-dark_sea
https://en.wikipedia.org/wiki/Linguistic_relativity_and_the_...
This is still quite clearly something different than being unable to see the different colors, though.
Their mental model, sure. The way they convey it to others, sure.
But you can easily distinguish between two colors side by side that are even closer in appearance than wine and the sea, even if you only know one name for them. We can differentiate between colors before we even know the words for them when we're young, too.