alt Hacker News> Oracle routing is a method for measuring the best theoretical performance by running the task through each model and then picking the cheapest correct option (the cost/performance ceiling).
Their "router" is an oracle reference point where they choose the lower cost model after running both and therefore knowing who passed the test. The cost savings part is only Fireworks theorizing what would happen if an equivalent predicting router exists. That's a big if.
Replies
andsoitis • today at 7:17 AM
> running the task through each model and then picking the cheapest correct option
What system knows what the correct option is and how does it know it?
➕ show 1 reply
The way they published this is baffling to me... surely you can try to implement some router and then see how well it does. Using an Oracles makes the whole writeup so much less interesting.