I'd say you're right about any given individual channel: the activation of a single voxel doesn't tell us much about all the fancy computation happening in that ~1 mm^3 of tissue.

But the pattern of activity of thousands of voxels across cortex does contain reliable information! And a decent amount of it too, at least in sensory cortices.