alt Hacker NewsI am afraid that this post is missing the biggest point.
Given a prompt (or task) how do I evaluate if it is a "simple" task that should be executed by a small model or if it is a complex one that may need a SOTA model?
Are you guys using heuristics? Which one?
Feel free to take a look at the docs for how routing decisions are made: https://role-model.dev/
When you use Pi with pi-role-model, Pi will include task and role metadata with its request; the role-model router runtime additionally holds benchmark and observability data, and a configured routing strategy. A composite of this is used to make the decision.
What you are pointing out is correct: making the actual decision and ensuring it is accurate can be difficult, which is why you need rich data as above, and also a model pool where each model is distinct, as I wrote in this post and in one of the comments below.