alt Hacker NewsI have mixed feelings on this (besides obviously being sad about the loss of a good person). I think one of the useful things about AI chat is that you can talk about things that are difficult to talk to another human about, whether it's an embarrassing question or just things you don't want people to know about you. So it strikes me that trying to add a guard rail for all the things that reflect poorly on a chat agent seems like it'd reduce the utility of it. I think people have trouble talking about suicidal thoughts to real therapists because AFAIK therapists have a duty to report self harm, which makes people less likely to talk about it. One thing that I think is dangerous with the current LLM models though is the sycophancy problem. Like, all the time chatGPT is like "Great question!". Honestly, most my questions are not "great", nor are my insights "sharp", but flattery will get you a lot of places.. I just worry that these things attempting to be agreeable lets people walk down paths where a human would be like "ok, no"
Replies
> One thing that I think is dangerous with the current LLM models though is the sycophancy problem. Like, all the time chatGPT is like "Great question!"
100%
In ChatGPT I have the Basic Style and Tone set to "Efficient: concise and plain". For Characteristics I've set:
- Warm: less
- Enthusiastic: less
- Headers and lists: default
- Emoji: less
And custom instructions:
> Minimize sycophancy. Do not congratulate or praise me in any response. Minimize, though not eliminate, the use of em dashes and over-use of “marketing speak”.
> Like, all the time chatGPT is like "Great question!".
I've been trying out Gemini for a little while, and quickly got annoyed by that pattern. They're overly trained to agree maximally.
However, in the Gemini web app you can add instructions that are inserted in each conversation. I've added that it shouldn't assume my suggestions as good per default, but offer critique where appropriate.
And so every now and then it adds a critique section, where it states why it thinks what I'm suggesting is a really bad idea or similar.
It's overall doing a good job, and I feel it's something it should have had by default in a similar fashion.