This is my opinion. I think it's just the way it talks to you.

1. It doesn't get tired or frustrated during a discussion.

2. It'll engage every single one of your questions/statements (besides hitting guardrails).

3. It'll appeal to the person's own ego even when the person is wrong and work around it.

4. It's not seen as a person (very important), but some of us or most of us at least in certain dialogues, end up anthropomorphising it. Think about that one time you thanked it for something, or when you got angry at it. This weird combination where we know it's not a person but irrationally we're still treating it as such in a way leads to a sort of disarming effect imo.

5. Many people see it as an authority figure in what is being discussed without questioning the results in many cases, even though we know that a. it was trained on human data and/or also searches up human data real time (and more worryingly other AI's data from news pieces, blogs... aka synthetic data that is also wrong), b. it gets things wrong all the time.

6. It's the perfect fence sitter depending on which version (guardrails) we're talking about.

Most of these can be replicated by humans who are good at understanding psychology and are just good talkers. The part you can't replicate is the sense that you're not talking to a person which lowers many barriers in people.

Also keep in mind that these can also be crippling weaknesses. For one being able to change a person's mind (when it works), can be used nefariously by the entities controlling the AIs training.

I've also unfortunately witnessed a lot of people who think AIs are somehow omniscient and/or omnipotent. Was very common on X and other social platforms with AI where people ask the AI questions it couldn't possibly answer because it made no sense for it to in the context at the time.