> Getting the benchmark tasks right is harder. I don't think there's any way to get around writing the task and its evaluation by hand.
This is a great tip and explains the main issue I had with my attempt at vibing a benchmark. I set up an arena so agents can make screensavers, then I judge the screensavers blind. Worked as an idea, but the prompt tuning was bad, agents would just build slight variations of the given example ideas. I'll have to try giving my prompt the human touch.