Anecdote:

I used to use LLMs for alternate perspectives on personal situations, and for insights on my emotions and thoughts.

I had no qualms, since I could easily disregard the obviously sycophantic output, and focus on the useful perspective.

This stopped one day, till I got a really eerie piece of output. I realized I couldn’t tell if the output was actually self affirming, or simply what I wanted to hear.

That moment, seeing something innocuous but somehow still beyond my ability to gauge as helpful or harmful is going to stick me with for a while.