logoalt Hacker News

ttultoday at 5:29 PM0 repliesview on HN

Fable seems to be a larger model. It costs more to run and does not seem superior for _typical_ software engineering work. But for work requiring raw intelligence, perhaps its size is an advantage.

On the DeepSWE 1.1 benchmark (IMHO currently the most relevant and least gamed SWE benchmark), the cost-benefit is clear: 5.6-Sol on xhigh achieves a slightly higher score than Fable 5, but consuming half the tokens and at about 1/3rd the cost.

But, on the Artificial Analysis intelligence index, Fable 5 appears to slightly beat 5.6-Sol, albeit at 3x the cost.

When I am coding, I send tasks to each model to get multiple opinions and it can be hard to predict which model will “win” because the results can be subjective. OP’s task is at least quantifiable, which is great. But many SWE tasks cannot be quantified so easily.