Price per 1M tokens is meaningless

Cost per tokens is as valid as price per unit volume of fuel.

Changing the fuel type, efficiency of your vehicle, driving distance, or driving conditions will all change how much it will cost you.

Fuel cost per unit volume does not become meaningless just because you are neglecting all of the other factors involved. That would be throwing away the only data point you have been using.

This is just asking for someone to amalgamate all of the factors involved into one simple, easy to game, index.

Cost per token doesn't say a lot, but "Cost per benchmark task" is also meaningless if your task is difficult enough that the cheaper model has no chance of cracking it.

I keep trying to convince directors and executives at my company to look past the cost per token amount but they refuse to do so. Those are the only things that actually give any sort of measurement of the monetary value of a token by these labs, and so its what many go by.

For example there's some benchmarks that show that Opus for any task that requires a higher than `high` level of effort, may have actually been cheaper to use Fable on low even though the cost per token is drastically higher

Similarly with GPT 5.5 vs Opus. They simply look at the dollar amounts the labs assign to each model and run with it.

But part of the issue compounds on the fact that there are many people who simply default to the smartest model/effort and don't actually vary their model per task. So in some sense I don't actually blame them very much.

In the context of local LLMs on limited hardware I've ran to the exact same conclusion: "tok/s" isn't the most useful metric when my personal North star metric, given my fixed hardware is: Model smart enough to execute my goals _in the minimum amount of time_.

Some models I tried (Mistral I think) had better tok/s, and roughly same billion parameters / scores on various benchmark... But they were _so_ verbose, that they generated many more tokens compared to a Qwen model of same caliber to answer the same thing.

So even though it had better generated tok/s, because so many more were generated, the clock time was longer.

And this compounds over mutli-turns: more generated token means more context used in the next turn (until some compaction or something runs)

As well as cost-per-task I think it's worth thinking about speed, especially in non-coding contexts that benchmark less cleanly

We've started trying to do some comparison videos to capture more of the UX vs speed vs cost stuff e.g. https://www.linkedin.com/feed/update/urn:li:activity:7479891... which one of my team did for my LinkedIn account (disclaimer: marketing)

(In this particular case Deepseek was way slower than GPT 5.5 but I think that's because it installed Libreoffice half-way through the task!)

An LLM is an extremely complex thing used for all manner of purposes. The hope that there would be some simple pricing construct that would map nicely to value provided is a pipe dream.

Pricing per token is at least reasonably straight forward. If you aren't getting value, you don't use the service. One doesn't buy a Ferrari and then complain that in their town Ferrari doesn't help them pick up women and hence it should cost less.

The Sonnet 5 comment is spot on. Even Anthropic's own graph initially showed lower performance at higher costs. Only thing I notice about Sonnet 5 is that it does appear to hand off tasks to agents more frequently similar to Fable, but of course nowhere near the quality of Fable. My guess is that Opus 5 will do similar but just isn't ready yet.

The variable missing from cost-per-task: which tasks shouldn't be hitting an external API at all.

Yup. I’ve been evaluating several on openrouter and find token cost meaningless for my work. I haven’t found a great alternative, though the “cost per task” he uses makes some sense.

I believe the future lies somewhere in between. I'm working on a hybrid application to reduce our company's token consumption. It runs on our data center's computing infrastructure and on laptops in our community. You might be interested; you can check out the code if you're interested: https://github.com/vfalbor/hibrid

Pricing based on tokens always seemed a little weird to me.“Tokens” was and still is an engineering concept. The fundamental unit of transformer encoding and decoding.

But I have a sinking feeling that many AI developers think “tokens” got their name from the same idea as “virtual tokens in a casino” which is more related to product pricing and business.

Related to this, for our use case, setting thinking to high instead of low made tasks complete faster and cheaper (Gemini 3.0 flash).

Other aspects are caching, often at 0.1X cost, where providers really differ in how efficient they are (Anthropic really good, Google not so much) and how chatty a model is (costing output tokens).

Price per token is meaningless for more reasons than this, because all of the provider monthly subscriptions price tokens _extremely_ differently than their per-token billing rates. It's stupid to look only at what you get when paying more than you need to for a given service.

This reminds me of cpu benchmarks vs actually running games and measuring FPS.

Well, not totally meaningless but certainly can be misleading.

cost per benchmark task is definitely interesting!

i've always wanted cost per prompt, but even that has too much variation.

tool use is another factor, every time the agent uses a tool the entire context is priced at cache rate on top. the same happens when it asks you for input.

Another important benchmark would be — cost per benchmark task using subscription tokens. Since most of us are using subscriptions and cost per token there is quite different from API costs.

would be nice to have these benchmarks so they can be run against models like the qwen family, gemini, etc.

[dead]

My advice to any CEO / individual - throw your hands in the air and bring it in-house. Yeah the performance can dip depending on what GPUs you can salvage these days but the uncertainty over price is almost nothing compared to the uncertainty over the effective use of AI. It’s not just coding (do I go partly agentic or all out Steve Yegge). This is all over the enterprise - do we parse every email, rewrite PowerPoints? Or just stop using PowerPoints at all. Do we throw LLMs at the mess of wikis and word docs, do we pretend that the policies no-one has ever read actually are how the LLM should think or is it how the work actually gets done - barely documented

The uncertainty of how to use this vastly vastly outweighs the price in a data centre - so buckle up, buy enoughbGPUs to experiment at a known cost and one day you will find the approach that gives you 10x returns - at that point pay any price per token but not till then