alt Hacker Newsi would love a deterministic program that can confidently make plans for lower-cost models like deepseek. ofcourse the LLM part wont be deterministic, but its a lot easier to measure quality like this. you could argue an AGENTS.md is this, but from experience its not enough to make non-frontiers act have a high success rate.
Replies
I've been working on a pi extension to do this, after frustration with getting the best current local models to stay on track, and just to deal with their relative slowness; gist is that it uses pi hooks to keep the model deterministically on track and beads-rust issue tracking tool to keep everything organized.
A SOTA model writes the initial prompt, and creates the beads issues; then a continuous iteration of plan(local) -> review (sota) -> implement(local) -> review (sota). Until the sota reviewer model is happy with the implementation.
And can mostly just let-it-run; e.g. overnight since local models with mac unified memory are slow.
Still early days, but have had reasonably good success with a Defender (1981) clone and now I'm having it work on a Prince of Persia clone (both Go/ebiten).
Note: I think this is only possible now because Qwen3.8-Flash-Next and 27b are incredibly good models.
I’ve had success with writing eg shell scripts that have a deterministic scaffold for the thing I’m actually trying to do, then call off to the agent for only the things that actually need it. For example I was trying to find a race condition flaky bug in my code, so the shell for loop ran the build N times, and called out to the agent to analyze the build logs if the tests failed, then the shell would ping me on slack when it was done, so the whole thing could run in the background.
I’ve been thinking more about how this deterministic + agents style could work, it’s kinda like the analogy of factories in the 1800s going from the central shaft to electric motors on each desk (where the central shaft in this case is a chat window, and the motor on a desk is calling the llm api from wherever you want just like a normal api call)
Of course, maybe in a few months the agents would just be reliable enough to do the shell part on their own too, but we’ll just have to wait and see
There are a lot of ways to slice the problem of getting the agents to complete a goal without getting lost, and the right solutions are somewhat problem-specific. For the projects I've done 35B Qwen is about the smallest that seems to make useful progress in a general purpose harness while 4B Qwen is workable with a task-specific harness. At the lower end the plan has to be traditional search/planner techniques in code not something the model has any control over, of course that limits the kinds of problems that fit. The high end coding models are perfectly capable of making a functional 1-off harness for those jobs so it ends up not being that bad to implement.
Fair — AGENTS.md is prose the model has to re-interpret every session, and that reinterpretation is exactly where weaker models lose the thread. Here the plan is parsed and enforced as structured data: tasks with declared dependencies and one prompt each, so the per-step job is smaller and the plan isn't up for renegotiation. Nothing in that needs a frontier model I just haven't benchmarked it against deepseek-class runners, and the runner is pluggable if you want to be the one who does.
I just can’t get excited about any of these meta-frameworks.
Doesn’t everyone get by now that any advance just gets rolled in to Claude and Codex a few months later, then the downstream competitors a few months after that?