alt Hacker NewsTesting it against Kimi is potentially skewing the numbers massively. Kimi has a number of quirks that requires behaviours that e.g. Claude or GPT doesn't.
Harnesses that are built around needing to work with "weird" models will need to deal with that, such as Kimi's tendency to get stuck in tool-call loops.
Harnesses built to deal with e.g. Anthropic's models primarily, do not need to deal with that.
Claiming on the blog that this gives Kimi Code no home field advantage seems like a dicey assumption. I haven't dug into the newest Kimi Code much, but the older Kimi CLI included several tools that were clearly specifically aimed at working around that behaviour - when I copied their checkpoints and "dmail" mechanism into my own harness, the performance with Kimi improved dramatically, but it made zero difference against Anthropic models.
That doesn't make the data worthless - it's clear you shouldn't use Clade Code to work against Kimi. But it does significantly limit the utility of it.
Afaik the reverse is also true. Some models are trained for specific harness setups so work better/more efficiently in that harness. If you give a model trained on a Read/Write/Bash tool setup a harness with only Bash where it needs to generate shell commands, I'd expect it to perform differently.
Related, there was a small debacle when Claude Code yeeted out a prompt change telling Opus to use Bash for everything. It was speculated it was an optimization to try to make it more efficient by encouraging it to chain a bunch of "tool use" commands into a giant shell command. However, they didn't communicate it clearly and it broke a bunch of setups that expected/hooked tool calls
There's also the (anecdotal?) tradeoff that dumb models tend to do better with smart tools and smart models tend to do better with dumb tools (smart tools -> easier to use but require more context usage for specs, dumb tools -> harder to use but more versatile and save context)