alt Hacker News> running the task through each model and then picking the cheapest correct option
What system knows what the correct option is and how does it know it?
> running the task through each model and then picking the cheapest correct option
What system knows what the correct option is and how does it know it?
Certainly sounds like a "P vs NP" style conjecture that shouldn't be possible in practice, save for certain generalizations, such as "this is a cybersecurity task, we know Fable will refuse (and score zero), so we just route to K3".