logoalt Hacker News

Havoc • today at 6:38 PM • 1 reply • view on HN

What type of tasks did you use for this? Evaluation?

Last time I did this I found it quite hard to come up with tasks that have clear evaluation outcomes


Replies

bertmuir • today at 6:54 PM

Have it compare to the real work you've done in a sample of your previous sessions. Agents are good at simulating completing the work following newer harnesses, heuristics etc. and comparing time, tokens, quality, pitfalls.