This reads very dystopian. You are not optimizing to understand people, you are optimizing to weaponize that understanding against them.
When you know what someone will buy based on exploiting their unconscious preferences, and you are paid to increase sales, you will do it. Especially if your competitors are doing it too.
And this happens at scale, invisibly. People never see the manipulation.
In any case, it is not useful for most people. It is useful for the people doing the deceiving.
It's marketing. That's how marketing works.
The tech is interesting and useful, no need for the scary moral framing.
The original application of the entire field of data science or ML is/was actually based on this paradigm of finding "unconscious preferences" (your words) and hidden patterns. How one chooses to deploy the tech should be judged on its own.
On the current trajectory of tool/data abuse where Palantir et al. are leading the way, this is very low on the sinister scale.