alt Hacker NewsOpen-weight models are going to completely shatter these forecasts. It takes a little more effort – right now, probably won’t be true in three months – but you can achieve the same at 1/10th of the cost.
Replies
fragmede • today at 10:38 AM
And 10x the headache. Money can be exchanged for goods and services, and people pay money to not have to deal with things. If you don't have the money for it, you pay for it in dealing-with-bullshit credits.
> but you can achieve the same at 1/10th of the cost.
For some tasks, sure. But not for all tasks. And for some tasks, cost per token is irrelevant if it provides real benefits that are oom compared to what you had.
Local models are indeed becoming "good enough" for some tasks, but there are still tasks that they can't touch. There's a recent benchmark for kernel writing. Fable wrote a kernel that provides ~30% more throughput per unit of compute compared to the latest Opus max / gpt max. Does it matter how much that session cost in terms of one session if you can take that kernel, deploy it on your inference fleet and "magically" get 30% more tokens served to your clients? There are companies that would pay millions for such a "leap". Because they can make more millions down the line.