alt Hacker NewsHandbook.md shows that long policy documents do not reliably govern agents
Comments
Any model with a good score on this benchmark would have a good claim on superhuman abilities. Humans are pretty terrible at being thrown a long policy document and being expected to follow it
And while we shouldn't anthropomorphize these models too much, I wouldn't be surprised if many of the core reasons for failures are similar. Working memory is a limited resource; you can only focus on so many things at once; reasoning depth is limited; and many real-world policies are not actually meant to be implemented in the same way they are written and have insufficient specification of edge cases
With humans, we usually do the equivalent of RLHF, both via "training" with simulated cases, and via feedback while on the job. You would never hand a newbie a 124 page policy document and expect them to correctly apply it on the first task, or to do it reliably in the first month
Yeah checks out with my anecdotal experience with Claude. It is pretty great at following instructions - for about 10 minutes, after which it seems to ignore things I told it before.
I have quite explicit and strong instructions (e.g. don't write massive comments, use existing functionality, etc.) in CLAUDE.md files which seem to get bypassed surprisingly quickly when doing real tasks. Yet if I tell it these things in a prompt during the task, it performs way better.
Result is I'm trying to resist adding more and more things to CLAUDE.md files which in some scenarios it does well but in other scenarios totally ignores and messes up.
Most people don't understand that 'agentic AI' is a completely synthetic, force fed capability by extensive Reinforcement Learning on synthetic domain specific 'agentic' datasets on post training.
If the LLM wasn't post-trained to adhere to specific handbook, it just won't work. If the LLM wasn't trained on an use case the lab decided was worth making a synthetic agentic dataset, it won't work as well as you want.
There's a reason the main agentic task LLMs excel at are coding tasks, it's the way of working of the creators, and they understand intimately the flow and can train for it.
I believe the true way will be able to easily fine tune models on your agentic use cases, but it would require a big company to compile a huge dataset on it's way of working and I don't think anyone wants to be the first.
In terms of long context, accurate attention retrieval from early tokens is just impossible, given the expansion of RoPE encoding for the positions, or in case of Kimi that don't use it anymore, as well as deepseek, early context is heavily compressed you lose accurate information.
If people spent more time studying about AI and how it works, they would realize that the default should be to one shot prompt your task with a big, cached system prmopt, with an user prompt that is just dynamic data, specified to the cheapest model that can do the job.
Unless you really can't do this given your problem, you should try to make a graph of well defined, step by step oneshot prompts, and THEN if your problem still can't be solved with that, then you start leveraging agents.
Despite this giving better results, and being more cost efficient, is evidently too much work then just letting the AI do all the work.
Long policy documents are also a problem for human agents. Without special training no one will retain 180 pages HR employee handbook, fire codes, OSHA safety rules, FCC regulations, the US legal code.
If the stakes are high, e.g, proceeding in ignorance could lead to prison time, people will favor inaction, even if the policy technically permits a corner case. If the stakes are low, people will completely override policy for the path of least resistance.
There was an article a few years ago called "Lost in the Middle: How Language Models Use Long Contexts" https://arxiv.org/abs/2307.03172
From my experience this holds true to this day. It was one of my core observations for similarity to the limitations of human working memory on "Engineering for Bounded Cognition"
This article to me also implies that there are some potential issues with large Spec based development flows, which I haven't been able to pin down lately.
Specifically, having agent implementation drift from the Spec.
> Failures follow consistent patterns: agents let a plausible in-environment request override the standing policy
I prefer this behavior in a lot of situations. A plausible request can also be a genuine one.
Alignment at the grain of each tool response is way more effective than a static system prompt. I try to keep mine under 500 tokens. Why bother fighting recency bias? It's pretty much what you want most of the time.
This is real. I was very angry that AI kept breaking the rules I wrote, so I asked Claude to crawl its own history logs. And then I found: every time after it breaks a rule, the chance of breaking rules goes up.
I feel this is like few-shot in reverse. Few-shot is supposed to be good examples AI should follow. But when it breaks a rule, we correct it, it keeps breaking, and this whole thing actually raises the probability of more violations.
I wanted to know if there is any difference between writing rules in the prompt, writing them in CLAUDE.md, or not writing at all. So I did some short tests before. I asked Claude to open brand new sessions, test different topics with the rules I want to apply. The result turns out to be: in a fresh session, no matter the rule is in a prompt or in CLAUDE.md, models (Opus 4.8, 5 or Fable) all follow it fine, across models. Even Opus 4.8, the one always violates rules in our conversations, does it well.
I suspect it's the long context that breaks rules. But simulating a long conversation experiment is kind of hard, I still haven't found a good way to test it. So seeing this paper now, it completely answers the question I was stuck on these few days.
Besides, something caught my eyes in this paper: sometimes the model does run the check by the rules, and it really finds the violation, but its narrative still insists on its original wrong output.
My current approach is same as everyone here: use a separate hook or post-check to fix things. Because if you let the model fix it during generation, its narrative or main generation part sometimes just rejects the rule error it found.
I noticed this behaviour a few months back, I think I was using Sonnet 4.6 at the time... I have strict rules about comments in the codebase, this all for personal projects, and the reason I restrict comments is to keep the token count low.
At some point between the model i was using and the previous version of it, Claude started inserting massive comments with references to tickets and other tasks. All this while having specific directives on the CLAUDE.md
Since then I resorted to developing my crapware as if I was the floor manager of a vehicle assembly line, and I have a few highly-specialized sub-agents running errands around the main session, but only ever taking care of a single concern. The main session builds with the knowledge contained in things like CLAUDE.md but the sub agents make sure things like the no/low-comments directives are either enforced, or factored into the final product.
LLMs don't really read documents like parsers do, from what I've observed, they behave like they first skim the document and find the section that they think are relevant to their task, they don't really try to read everything and hold it inside their context.
The solution to that is pretty simple: get to the point, list all the requirements that they must do complete for that task, constraints on what they must not do, and optional recommendations for them to follow. Leave things that they can figure out on their own out of these documents.
Sounds about right
For Claude, I used a UserPromptSubmit hook running inject_rules.py which reads RULES.md from the disk and prepends the whole thing to every prompt. That helps the rules to stop fading as context fills because it is reinforced every prompt.
Sure, it uses tokens slightly faster in the prompt, but I find it reduces overall token use, you can use it with pro, but of course, nothing works 100% of the time, but it's better. Emptying the memory helps too to avoid Claude making up stuff that messes with how I want it to act.
The general gist of inject_rules.py is :
RULES_PATH points at RULES.md
reads it with encoding='utf-8-sig' so the BOM is stripped
wraps it in a JSON object — hookSpecificOutput.hookEventName = "UserPromptSubmit", additionalContext = a preamble plus the full rules text
the preamble is the line you see above the rules: rules are in force for this turn, run rule 33's five tests before raising anything unasked
prints that JSON to stdout, which is how Claude Code takes it in
on OSError it returns 0 silently — if RULES.md is missing or unreadable, nothing is injected and the turn proceeds with no rules
I absolutely believe it.
Codex has been pushing things to my main branch all week despite me repeatedly telling it not to and adding to my AGENTS.md very clear instructions for creating feature branches and putting up a PR. It keeps doing it in spite of all that.
I'm probably going to need to enable branch protection on my personal projects... What a pain.
Evoking Gödel via GPT, with the requisite em-dash: "No sufficiently expressive formal governor can be simultaneously consistent, complete, and able to prove every relevant proposition—including all propositions about itself."
But "Read AGENTS.md, including the middle" sure helps.
What really helped me was to run any .md/prompts through an LLM to find contradictions, duplicates or ambiguities, repeatedly. That led to agents much better following instructions.
Control vectors for the win? It feels like the way to fix this is to pull out a control vector immediately after processing the handbook and use that to steer later inference. That should stop the drift over long distances, but it doesn't guard against the handbook itself being too big.
For my own (rather idiosyncratic) harness I've been experimenting [0] with "compiling" long markdown specifications into small executable logic programs. It's too early to tell for sure, but I believe that this approach does have its merits when you want some guarantees about how your agents behave for longer tasks.
Policy documents do not govern agents at all. Conformance is distributed and completely unreliable.
How many times have you told an agent not to do something then had to correct it?
You must always flip the frame. Objective analysis is way better with llms than steering via skills.
This is just a small example of why "loops" became popular for a minute and now it is "graphs"
I don't understand why a model has to follow the instructions. Don't get surprised when it shows its true color. Plus, do users (not the researchers) really check whether the response follows the instructions?
Attention vs. Consistency
When "performance" breaks down over long lengths, one could attribute it to a lapse in attention, but one could equally suspect inconsistent instructions.
The fewer instructions and conditions that need to be simultaneously met the easier it is to comply, but with more and more instructions one is bound to introduce internal inconsistencies within the instructions.
Confirmed. These are autistic geniuses with ADHD. Handle with that care.
Opus 4.8 (max thinking) scored highest and Grok 4.3 lowest
It's hard to understand what's going on with Grok. It's like it has capabilities in a theoretical sense but maybe the training is so focused on being in x.com/grok.com with the web search tool enabled for "is this true?11" type queries that with any API type usage with document workflow instructions, tool use, code gen etc it completely falls over
Why would anyone think that models optimized for efficient context management, giving much more weight to a short sliding window, would attend to distant, heavily diluted tokens?
Plus the model's capacity to take more context into account and actually integrate it to the output is simply limited by the number of activated parameters. If you give it a playbook, you are forcing to choose it between attending to the playbook and the task at hand.
If you want to force it to work step-by-step, you need to present the steps one-by-one. Ideally with rules for the current step at hand and maybe relevant input again, depending on overall task size.
Why did you think models love to re-read files before editing them? It increases recall quality and thus edit precision and thus benchmarks.
Dealing with agents/LLMs based on "instructions & guidelines" has taught me - nothing (un)reliably governs agents other than agents themselves or (rather i.e.) their motherships (assuming they can and they intend to). Or if you add ton of local tooling.
One question I'm asking myself is: can LLMs be bored?
This is a problem with soft rules and LLM in general, and it makes sense that it gets worse on complicated tasks with long context.
Glad to be able to put some numbers on it.
simonw has been correctly asserting that there is no deterministic way to prevent hallucinations. I’d argue that this obstinance could be the seed of a strong argument in favor of their capacity to eventually coalesce some form of consciousness
I'd be very interested to see a comparison with actual humans on the same benchmark.
In my experience, the more structure you enforce on models, the worse they perform and the less they actually do what you want.
LLMs are not people. Employee handbooks, like all other social rules, are not self-executing; they depend on the fact that human beings are innately social animals.
This is why the future is in deterministic static analysis and policy-as-code frameworks.
Policy-in-English? Model implicitly complains that it's TL-DR.
Ask the model to write code that checks your policy, then add that code behind a simple validation hook (e.g. "check your work by running 'just validate'") that the harness knows to always run after changes? It suddenly becomes the most law-abiding citizen ever.
I got pumped seeing the OKF format from Google (which is just a standardization of wiki pattern) but quickly realized it could not yet combine many subtle concepts together in an efficient way.
Hard to take this seriously when it has all the hallmarks and annoying tics indicating it has been written by Claude.
This is what spec driven development tries to solve.
Multiple rounds of generating small contacts documents that grow from the original idea , trying to keep each slice small enough to process for a human to approve/disprove .
Eventually it leads to a long list of tasks grouped by functionality. You start a new context and the orchestrator agent dispatches tasks to sub agents with a limited amount of information provided to each sub agent.
Also should have adversarial review and approval gates with other agents and roles.
Having only dipped my toes in generative ai recently I'm surprised how small even a 1mio window is. A semi serious project can take several session in one sitting. Especially as degradation sets in waay before the window is full.
Well no shit obviously. Just cause you tell the random text generator to follow some rules doesn't mean it will.
[flagged]
[flagged]
[flagged]
[dead]
HANDBOOK.md is a benchmark for long-context agentic instruction following, modeled on how enterprise employees follow company handbooks in their day-to-day work. Each task is a unique RL environment with internal tools and external MCP servers, spanning five enterprise domains: Finance, Medical Billing, Insurance, Logistics, and HR.
The prompts reflect the actual jobs enterprise workers perform every day. Each task drops an AI agent into a live company environment, requiring them to cross-reference an extensive, multi-section handbook against a cluttered inbox, a multi-channel Slack workspace, Jira queues, and a stack of files (spreadsheets, PDFs), and working out both what to do and what the handbook forbids.
This is a problem with long context models. To put it as simple and as bluntly as possible: just because they claim you can use 1M tokens in your context doesn't mean its true and you should do that.
Due to extreme quantization of models and the context's KV cache, and also just really shitty samplers provided to the user (hell, most are just getting rid of sampler knobs altogether), this problem will absolutely continue.
Want it to go away, almost like magic? Local inference. When its under your control, and no longer being forced to hold it wrong, all of the common LLM defects will go away.