In my experience, Kimi just tends to think a lot, with the main thing that takes up a lot of space is it constantly second-guessing itself. I've watched it do paragraph after paragraph of "Wait, actually..." while it stumbled and used a ton of tokens on one small detail of what it was asked to do. Though I also gave GLM 5.2 a task to port some JS code to Python to test it, and in my experience it doesn't second guess as bad as Kimi does, but it really did there. It kept doing web searches and second guessing tons of tiny little things, using $0.25 of API spend in total to port about ~50 lines of JavaScript. It did produce an error the first run, but on second run it gave me a program that ran.

I gave Claude Code/Fable the same task and it took significantly less time, but also stumbled on the same error as GLM. I didn't have it fix it though. I was mostly interested in timing differences.

I do like open models where I can, but I'm really hoping they get trained to second guess less. Or maybe I just need to prompt them differently. I'm not sure.

We really need to stop using $/M tokens as the pricing benchmark. I've found that the number of tokens used tends to be a bigger factor than the listed per token price. The cost per task vs. intelligence curve is really what you care about, and in my estimation Chinese models are just not there. They are focused on benchmaxing and getting the highest raw score they can, rather than efficiency.

Earlier today I made Claude code implement a feature with fable. It worked roughly 60 minutes and used around 30% of my 100€ subs 5h sessions.

Then I typed /code-review in a second terminal/clean session after the analysis was done (no code changes) the usage was 99%. I then asked it to write that into a review.md so I could restart from that the next day. Sadly the last % wasn't enough for that.

Ymmv, these models behave very differently with no discernable reason. Usually reviews(even with fable) take like 10-20%... Yet suddenly you get it to burn through 65-69% in 15 minutes or so

> Subscription usage limits are hard to measure as none of the providers tell you directly what it means in terms of tokens or anything else you can easily compare

AI subscription pricing is so goofy. You get some amount of usage that varies by models, is measured by opaque token usage, driven by how many tokens the (usually) vendor-provided interface (or model itself) wants to use. Then your usage is limited by time opaque time windows.

Kimi K3 only supports "max" reasoning effort right now, but they plan to enable other levels soon [1].

When I looked at traces from benchmarking, I saw a lot of backtracking and uncertainty while reasoning ("wait, but..."). This also happens with GPT 5.6 and Fable with xhigh/max thinking, albeit to a lesser degree.

I think that explains part of the token inefficiency. Hopefully it will improve with lower reasoning effort settings.

[1]: https://platform.kimi.ai/docs/guide/use-thinking-effort

Yeah I've noted this behavior with best in class open weight models. They said K3 would have token efficiency improvements and I was hoping especially solving the thinking loop issue that plagued K2.x but even if this release helped somewhat, it looks like we still have a long way to go here... I'm not sure what's up here but I suppose lacking finetuning quality.

What OpenAI in particular have done with reasoning efficiency in the past few months since ChatGPT 5.5 is nothing short of remarkable. It's overshadowed a bit by the benchmark game and the Fable hoopla.

Now is the time to focus less on token cost and intelligence, but tokens to solve a particular set of tasks in closed benchmarks for a variety of categories.

What is the use of grand intelligence if it either costs you a kidney or can't complete at all within a token budget? Even if there are niche uses where you truly want "maximum power" above all, we need to at least more severely penalize such models versus those that does it just as fine within a tenth of the token cost.

I'm aware of some benchmarks at the Artificial Intelligence site, but CLEARLY we are not focusing enough on these today and still leaving the fun surprises to the users.

Absolutely do not pay for the kimi plans thinking they will be cheaper. If you sign up with a Chinese phone number, you can get the same plan for 200 yuan instead of 200 usd, it also only accepts Chinese payment methods iirc. So the plans are really made for Chinese userbase.

Is Kimi K3 subsidized as hard as the other models out there?

Do models know when they're being benchmarked? I also used K3 briefly and noticed it spent a very long time thinking and obviously a lot of tokens.

However I've seen some benchmarks say it uses fewer than fable which hasn't been my experience.

Aren't they still locking reasoning to "max" pending adjustments to support shorter reasoning levels.

It would be really interesting to redo the public benchmarks for kimi k3 but token normalize the costs. Ok so maybe k3 beats fable on terminal bench, but how many tokens did it use?

> tried Kimi K3 on a task I've done with every other model I use regularly and found it chewed a lot longer on the problem and ate up almost the entirety of a 5 hour usage limit on their $19 plan

ArtificialAnalysis puts Kimi K3 just below DeepSeek v4 & GLM 5.2 in token use per task, which is about 2x to 3x more tokens than Grok 4.5: https://x.com/ArtificialAnlys/status/2077832879187620192 / https://archive.vn/zBbFi 2 other open weights MiMo v2.5 & MiniMax M3 are comparatively thrifty.

> Subscription usage limits are hard to measure as none of the providers tell you directly what it means in terms of tokens or anything else you can easily compare

I always put my coding subscriptions (that allow it) through "AI gateways" (Cloudflare & OpenRouter are free) which help track token use.

In my experience, Kimi & Qwen Cloud have opaque & restrictive limits, their "credits" drain faster. I now make it a point of subscribing (directly [0]) with providers that are transparent like MiniMax, DeepSeek, Xiaomi, & Z.ai.

[0] OpenCode Go, Cline, and AtlasCloud have generous limits for open weights, otherwise.

With the obligatory disclaimer that I’m impressed with what open weight models can do, I have the same experience with all of them.

The benchmarks come out and say they’re as good as Opus from N months ago, then I use it for a complex task and it doesn’t work as well as Opus from N months ago did when on similar problems.

There’s a real wow factor when you get an open weights model to do amazing things, but in my experience the gap to the frontier models has always been bigger than the benchmarks would lead me to believe.

There can be a lot of value in having the cheaper open weight models for chewing through lower complexity tasks (non-programming in my primary use case) at a cheaper rate than OpenAI or other frontier API costs. Even with those I can measure bigger gaps to the frontier models than the benchmarks suggest.

If the benchmarks aren’t being directly gamed, there’s at least some selection happening where training data or model structures are being picked in ways to maximize public benchmark performance. All of the labs know there’s immense value in having good benchmarks to show for your model because most LLM consumers are picking based on lab provided benchmark charts, not running their own evals. Running your own evals is hard and expensive.

OpenAI measures token efficiency. Look at the API cost charts in their announcement: https://openai.com/index/gpt-5-6/