logoalt Hacker News

scosman • today at 7:41 PM • 0 replies • view on HN

We need this benchmark for non-coding harnesses. Harnesses with custom knowledge, custom tools, specific guidance, etc. A ton of the task-specific performance comes down to the skills/tool-set.

I've been building a tuner that tunes the harness (prompts, skills, tools, etc) instead of the weights of models: https://github.com/Kiln-AI/harness-tuner -- Might try to publish some "untuned" vs "tuned" comparisons, the numbers are dramatic.