I have thought about this before as well. Like maybe what I see as red you see as purple but since we have always been taught that what we both see is red to both of us it is red.
I am however leaning more to the belief that typically we all see colors the same. But it is one of those things that could never be proven.
Another interesting thought that comes to mind speaking about color perceptions is I recently read an article or post I honestly don't remember where that discussed what do blind people see like do they just see blackness all the time. According to what I read it claimed that people born blind don't actually see a blackout picture they literally just don't perceive anything. I think for most it would be hard to imagine nothingness but I could accept that as a true fact.