Yes exactly. I'm intentionally using "color" as a perceptual thing, not as a physical thing. If we are talking about a color model, then it needs to model perception. As such, RGB, as a predictor of perception, can often fail because it doesn't account for much more than what hits the retina, not what happens after. For one, it lacks spatial context - placing the same RGB value with a different surround will feel different, like in the example above. But if you had a real color (as-in, perceptual) picker in Photoshop, you would get a different value.
It's excellent at compressing the visible part of the EM spectrum, however. This is what I meant by stimuli encoding.
Still not seeing why you claimed color is definitely not something you can plug into a 3D model. We can, and do, use 3D color models, of course. And some of them are designed to try to be closer to perceptual in nature, such as the LAB space like @zeroq mentioned at the top of this sub-thread. No well known perceptual color space I know of, and no color space in Photoshop, accounts for context/surround/background, so I don’t understand your claim about Photoshop immediately after talking about the surround problem, but FWIW everyone here knows that RGB is not a perceptual color space and doesn’t have a specification or standard, and everyone here knows that color spaces don’t solve all perceptual problems.
I find it confusing to claim that cone response isn’t color yet, that’s going to get you in trouble in serious color discussions. Maybe better to just be careful and qualify that you’re talking about perception than say something that is highly contestable?
The claim that a color model must model perception is also inaccurate. Whether to incorporate human perception is a choice that is a goal in some models. Having perceptual goals is absolutely not a requirement to designing a practical color model, that depends entirely on the design goals. It’s perfectly valid to have physical color models with no perceptual elements.