alt Hacker NewsKinda funny that their "cost-vs-performance" chart looks the same as the one for Composer 2.5[1], except that it includes Composer 2.5 at a completely different spot.
What are the chances that CursorBench ranks Cursor's model highest, and Cognition's bench ranks Cognition's model highest? Both are to be RL'd from Kimi as a base model, BTW.
I'd posit that it's not deliberate deception, but for both companies their training data and benchmarks come from the same dataset (Devin/Cursor interaction logs) so they naturally overfit.
Replies
Good observation.
I actually started typing the same point that the chances are actually high because of train/eval overlap then realised you answered your own question with that same observation.
It is interesting though!
Perhaps in some way this means we should decide which eval set aligns best with our taste?
Back to the blog post. This is an excellent write up of an excellent technical achievement.
I have a lot of respect for the Cognition/Devin (always "Windsurf" to me) and Cursor teams.
I found it interesting - but justified - that they referred to themselves as a foundation lab rather than a dev tools company.
I think it's also telling that they left out the usual hallmarks of the Pareto distribution: GLM 5.2, Qwen 3.7, Minimax M3, and Mimo 2.5
https://arena.ai/leaderboard/code/webdev/pareto