logoalt Hacker News

protastus • today at 7:41 PM • 0 replies • view on HN

The AI verification is not "handled structurally".

The vast majority of my tokens don't go into the authoring -- they go into review iterations. And the problem is that many reviews don't converge, and even the ones that do, can do it slowly, with drift and leave a residue.

Reviews don't converge when the premise is flawed. You can put tripwires to try to catch this, but it's not a robust process. Agents will generally try to act on a flawed premise, especially when they can't foresee the long-term consequences of the axioms under which they are operating.

Reviews drift because the author and reviewer influence each other to depart from the original problem statement. This is also correlated with unbounded scope creep and slop. Again, one can put tripwires to detect this and escalate but agents are extremely creative at finding new ways to ruin your day.