I expect doing this well requires the underlying system to somehow have a clear differentiation between whether a color is mapped to the natural world or not.

In other words, colors assigned to good/warn/error icons can be shifted almost arbitrarily as long as they are consistent over time for one person, but it backfires if you're doing the same shift for colors in a picture of a tree's spring/autumn/fall leaves.