alt Hacker NewsSomething related I've been thinking about lately is that one of the biggest problem with LLMs is their seeming inability to say no. Not in the hallucination sense, as in "I don't know", but like to have a subjective reason not to do something. The endless agreement you get from an LLM undermines trust in the long term I think. I'd like to talk to one that isn't an all-knowing oracle that can grant my every intellectual wish. (Or maybe what I'm asking for is just... a human, lol).
Replies
It's interesting because i'm kicking the tires on the top tier stuff for a month (because it's expensive as fuck but I need to know where the ceiling is).
I have actually gotten "hey i don't think this is a good idea, here's why" as feedback from at least Opus. It WILL still do it if I just demand stupidity (and hell i've been right, which is another topic entirely) but it has given me more confidence this can be a useful tool in the right spots.
That said I probably don't need the top tiers (metrics at least confirm that) and I'm guessing that's specifically because I was working in coding. Most were worded in a "is this a good idea" framing which probably helped, but at least once I said 'lets use this library/method" and it gave a decent argument on why that was basically redundant without prompting.
I still struggle to see the price point panning out.
I think that is an issue. Also, the ability to quickly build any idea might not be such a great thing. Not only do we probably all prefer things of quality that were made with care but some ideas also just shouldn't be built.
Over the last 3 years I've seen projects where I thought, pretty obviously that's a bad idea. But, because LLMs don't say no and can just be pushed to build it anyway, the people building them might never learn that or learn why.
It's nice to be able to have a quick prototype or mvp. But if we never hit friction or something not working out, we never learn or have to come up with a creative solution.
Now, the LLM might seem incredibly intelligent (relatively speaking) and also creative but let's not forget that all is based on its training data. I simply don't believe it can ever be omniscient or that the companies training it are careful enough when doing so.
I’m using ChatGPT and started to notice that lately it answers my prompts starting with „Yes” even if my question was open. As if the first token gets injected and the LLM is left to finish the response in a sensible way, often ending up with some form of „Yes, but not really”.
I've been wondering whether that is a feature of the foundation model or whatever finetuning they do on top. I remember this from the earliest versions of (pre Chat-) GPT I've been using, which would suggest it's a feature of the foundation model. But I don't really understand why. Something that's been trained on StackOverflow and BB forums, among other things, should have seen a ton of examples of answer refusals.
Agreed, it is abolutely an issue. It is quite difficult to find an optimal solution to some problem when every considered new idea is ”definitely the right shape”.
> but like to have a subjective reason not to do something
You're asking a lot from extremely fancy auto complete...
The model providers could randomize the system prompt to make it say no 2.36% of the time, automatically tuned up or down depending on user feedback.
You've hit on an important insight.
People smarter than me have a habit of getting me to see things without telling me. They ask the right questions.
LLMs, incidentally, respond in a similar pattern in my experience.
This is an active area of research to inject humility into llms in order to create some kind of knowledge boundary. You can look this paper from nouswise https://arxiv.org/html/2604.17843v1 and the product build on top it to try the humility.