I’ve thrown my agentic workflow at Terminal Bench 2.1 and it found a bunch of issues (aka failed tests) because the prompts are “bad” and verifiers are overly specific.
As an example, there’s a task that asks to make a MIPs interpreter to run Doom, and save a frame at something like /tmp/frame.bmp
My spec-driven flow was like “this is useless, let’s record frames like /tmp/frame-N.bmp”
alt Hacker News
I’ve thrown my agentic workflow at Terminal Bench 2.1 and it found a bunch of issues (aka failed tests) because the prompts are “bad” and verifiers are overly specific.
As an example, there’s a task that asks to make a MIPs interpreter to run Doom, and save a frame at something like /tmp/frame.bmp
My spec-driven flow was like “this is useless, let’s record frames like /tmp/frame-N.bmp”
Instant fail.