This is true, and illusionary at the same time.

While our precise perception of red may not match, the interplay between colors is such that people perceived them to go together, or clash, etc, in a somewhat consistent fashion.

This means that, over the general population the perception of color is very similar from person to person. Ignoring genetic defects.