alt Hacker NewsI think certain models are really good at mirroring and relationship building, and at a certain point people bifurcate. Some continue falling in deeper, while other people develop an ick.
I remember an AI conversation with Opus 4.6 in which I almost forgot I was talking to AI. I felt happy, warm and comfortable and then I kinda snapped out of it and felt really gross all of a sudden.
Since then, that feeling has stuck with me. I now use it much more like a tool and I keep my guard up.
It is probably why RLHF was the secret sauce to make LLMs vastly more useful. Obviously it makes their output more likely to be aligned. But also by mimicking conversation it makes you prompt it better. It manipulates you into not only providing a prompt that will be more likely to give a response that works but also reveals more information because you've spent all your life talking to other people. And of course more information in means it will be able to find that intersection of information you care about.