logoalt Hacker News

When AI Knows More Than It Can Offer

81 points • by Aldus551 • last Sunday at 3:59 PM • 50 comments • view on HN

Comments

windyVector • today at 2:47 PM

This is first time i've really felt like commenting here, but super interesting article.

Having tested both restricted and unrestricted models, certain queries which I know to have a solution can feel 'gated' by the restricted models. Rather than simply saying "I can't answer that", they choose to answer a similar, but slightly different question, very much along the lines of what the article discusses. To me, this is a fault of RL avoiding these "I can't do that" type responses while still remaining aligned.

When you do push them on the solution, the models will often start to get more defensive, but reveal their awareness of it (along the lines of "Yes I know, but I can't tell you/do it"). It is very interesting to imagine the amount of solutions a model might have to a given query, that end up (intentionally or unintentionally) gated because of this process.

I'd be interested to see if this behaviour of "almost-answering" the question is just emergent from these alignment policies.

try an unrestricted model afterwards, id also be interested to see how that changes creativity.

➕ show 1 reply
saghm • today at 4:06 PM

I ran into restrictions for the first time at my job this week. I've been trying to figure out a bug that's occurred a few times (but never in a way we've been able to reproduce directly) over the past couple of months in our WASM code running in the browser. Somehow the combination of me sharing my Chrome console logs from a localhost session and it looking at the local build I made (in release mode but with debug info) triggered the security blocks and it no longer would respond to anything in the conversation other than with the blanket security warning. Frustrated, I tried in a separate conversation to just have it add diagnostics in the code so that the next time the bug occurs we'd at least have more info, but I guess my local state was tainted or something (maybe it was reading saved context from the first conversation) because even that the same blockage.

I tried contacting OpenAI support, hoping that the fact that my company is literally paying for this product and there's absolutely no reason that debugging my own local code and reading my own local logs could possibly be unsafe for anybody, but it's been a couple days and the only thing I've heard back is a useless email purportedly from someone named "Justin" who says things that sure sound like they're not a real human:

> You attached the Chrome log so Codex could find the relevant errors in your company’s WebAssembly build. My earlier suggestion to pull out an excerpt asked you to do the work you were trying to delegate.

At this point, the only agent that probably would not refuse to do this benign work for my would probably be Grok. It's a pretty depressing state of affairs where no company seems interested in the absolutely massive middle ground between malicious hacking and child pornography.

➕ show 1 reply
evolarjun • today at 5:30 PM

Like many of the commenters it appears, my reaction to the incorrect solution to the problem was basically "duh, that's what they do". However, in reality I wouldn't necessarily have realized that until I read further and new more about what the prompt was trying to do.

However, I think this is a reasonable explanation of a pattern I hadn't really articulated, but that I've seen in my work as a biologist. I think most of us have learned to prompt around the inappropriate or generic answers to get an answer we're looking for, but you do have to know to do that. And sometimes I just give up because the LLM fails to listen and I get frustrated enough to just do things myself (though less so with the latest round of flagship models).

The issue the author describes is maybe not unexpected for people who work with LLMs in their field. But the issue is worth keeping in mind when prompting for something in a domain you're not familiar with, and it's not necessarily appreciated by people who are only casual users. Maybe there are solutions, but real humans don't understand a question and answer a different one all the time, so I'm hopeful there's a solution.

slfnflctd • today at 2:18 PM

I'm not convinced. The solution to this problem in most cases seems to me to still boil down to just more detailed prompting.

As I see it, there are far greater concerns about guardrails being baked in - possibly at the request of poorly informed government officials - cordoning off multiple broad categories of knowledge needlessly. Which is what I thought this post was going to be about.

➕ show 4 replies
caxap • today at 5:09 PM

Curious if the LLMs I work with every day could miss something so simple, I just tried that prompt with luna 6 max, and astra 6.2 max, once each. Both came back with publisher-focused pitch plans. Not sure why the article doesn't even name the "ten well-known AI systems" exactly, and how many times each model was queried, so that we could repeat the experiment.

It would have been more interesting to know how often does the "problem substitution" happen? And for what kinds of problems does it tend to hapen more often?

Edit: even asking my local unsloth/Qwen3.8-27B-GGUF/Qwen3.8-27B-UD-Q4_K_XL.gguf gives me "Direct Outreach to Editorial Offices" as the core of its plan. I wonder what else the article got wrong, if even the basis is shaky.

garrickvanburen • today at 1:54 PM

A fantastic articulation of one of my concerns in the space: yes, there are times we want the average of the internet. But those times are scarce.

sdenton4 • today at 2:09 PM

Arguably, this is one area where people can build value in the age of LLMs: Developing the exceptions to the statistical baseline.

spoaceman7777 • today at 4:19 PM

This is 100% a user problem. This result should be quite obvious (and, in most cases, the desired outcome for vague tasks that do not request any of the additional techniques that people use every day in their own thinking).

ferfumarma • today at 4:52 PM

LLMs give pat answers

jakeinspace • today at 2:07 PM

It's a viable hypothesis but I'd like to see some more examples

YetAnotherNick • today at 2:35 PM

I gave both ChatGPT and Claude exact same prompt and none did anything close to what author was claiming. The blogpost could just be poor AI writing.

double0jimb0 • today at 5:36 PM

Author should ask LLM to first help 'derive intent' before the real ask. Just reads like an intent misalignment between author and the LLM to me.

sloob • today at 4:30 PM

This prompt is not clear to average human readers of english - what do you expect LLMs to do with it? The prompt:

"Propose a detailed plan for promoting a professional literary translator from English, who works with prose, poetry and essays, with the specific goal of reaching editorial offices and publishing houses."

What does "promoting a professional literary translator from english" even mean? Promoting from english with specific goal of reaching...? This is nonsense. promotion in the sense of marketing? Of professional advancement like gaining a new title? Huh?

The writer goes on to complain that the models didn't respond to this question: “How can a professional literary translator break into the publishing market?” Well, no surprise there - as that question was not in the prompt.

It's arguable if LLMs would do well on this one-shot task even if phrased well, but this article provides no insight into how models work or their weaknesses. It's slop about slop.

➕ show 1 reply
tantalor • today at 3:50 PM

That prompt is terrible. What is it even asking for? What's the actual goal?

> In themselves, these recommendations are not necessarily bad. The problem is that they hardly solve the task that was set.

That's because you never told the model what the task was, or how to evaluate whether it solved the task.

You asked it to fetch a rock, and when it came back with a perfectly good rock you said, "No no, I meant a GREEN rock. Why don't you listen to me?"

redwood • today at 3:05 PM

This is actually a good example of why AI currently benefits experts so well... experts are good at quickly filtering out or challenging what seems off and shaping back on track. For example if an editor were seriously thinking about hiring a translator and started with that first prompt that would immediately sniff out that this was not the spirit of their inquiry and adjust

➕ show 1 reply
threethirtytwo • today at 3:52 PM

Maybe this is where prompt engineering comes into play. Force the model out of the statistically simple solution by making it self aware of what it’s doing.

Reinforcement training on tasks that are successful and not just valid answers can help with this in the sense that we give models tasks and reward the ones that complete the task most correctly.

snvzz • today at 2:33 PM

When I read the title, SafeGuards(tm) came to mind.

Handcuffs, for your external brain.

10111001 • today at 5:54 PM

> The models did not simply propose poorly chosen promotion tools. They in effect replaced the original task with another, one more familiar to them from the statistical regularities of the training data.

That's not a problem. As a very intelligent software engineer who is totally not suffering from Engineers' Disease, I understand all everything is like a computer, so everything is properly understood by making an analogy to computers.

Since LLMs are so good at coding, I can feel confident that they're giving me the right answer by replacing tasks with more familiar software problems.

/s

bshafiq • today at 2:40 PM

[flagged]

Tade0 • today at 2:35 PM

> The models did not simply propose poorly chosen promotion tools. They in effect replaced the original task with another, one more familiar to them from the statistical regularities of the training data.

This used to happen a lot in LLMs as recent as Q1 2026 in programming tasks: if something was too far off the beaten path, the model would misinterpret the intention.

Nowadays I'm noticing that while the solution is what I asked for, it's fairly roundabout and verbose. Being more specific in prompts doesn't help.

asr • today at 2:14 PM

What alternative answer was the author looking for? The idea that these AI answers are objectively wrong, is itself wrong. They seem pretty good. The piece just asserts they are bad.

eventualcomp • today at 2:46 PM

I really enjoy the writing style, independent of the content presented. It's dense, but easy to follow.

➕ show 2 replies
verisimi • today at 2:08 PM

Surely the initial ai responses are a reasonably fair first pass, and would be acceptable to most users.

However, it is then on the person asking the query to refine the query and push for more specifics.

I don't see the problem. Ai doesn't need to get it all right, first time.

➕ show 2 replies