alt Hacker NewsAn agent doing a task even with multiple back to back calls like normal without an example is zero shot. An agent doing a task with 1 example is one shot. An agent doing a task with a few examples is few shot. I don't think you are correctly using these terms.
The multiple back to back LLM calls are done on accumulating context, so if there is a sampling error it could throw the entire session out of whack, because LLM's build on the previous context.
It's actually meaningless to argue, one could simply sample more than 1 times and let the numbers speak for themselves.
Replies
That's not true. An agent in a loop can test itself, review, verify and iterate as much as needed. That's one of the primary reasons more capable models tend to have a higher success rate.
I don't disagree that multiple tests increase confidence, but it's not correct to argue that an agent in a loop harness is equivalent to oneshotting
> An agent doing a task with 1 example is one shot. An agent doing a task with a few examples is few shot. I don't think you are correctly using these terms
This is a different thing. Yes, giving multiple example is called "few-shot prompting".
But one-shot vs few-shot benchmarking is different. In this context "one-shot" means "pass at 1 effort" as opposed to "multi-shot". In the literature this is called "pass@k".
Anthropic has a good explanation here: https://www.anthropic.com/engineering/demystifying-evals-for... (search for "pass@k").
In this discussion we are discussing pass@1 (single shot) vs pass@(k>1) (multi shot).
> The multiple back to back LLM calls are done on accumulating context, so if there is a sampling error it could throw the entire session out of whack, because LLM's build on the previous context.
This isn't really true. In an agentic loop the LLM can correct itself via in-context learning.