>nobody is going to be doing 17 benchmark passes on a $50-100 task

Hopefully we'll see more of this as big companies try to optimize token usage where the cost of benchmarking is dwarfed by the potential savings across the org