I think this would make sense for consumer hardware, not for AI companies.

AI companies constantly update/change stuff, new models come out, new requirements, etc.

But if you ship an "ai-powered" dishwasher, it can come with the chip built-in to do computer vision and precisely target each spot, and will be sold as-is with no updates.

I think the real value here is not as a customer-facing agent/chatbot but for for automated processes. Think of all the companies out there that have LLMs doing simple tasks like categorizing customer feedback emails. For such tasks, you don't gain much from better models, so if you could run it 10x cheaper on a slightly older model, it would absolutely be worth it. Pretty much any place people are currently running a flash model could benefit from this since they're already deciding that speed+price is worth using a less capable model.

I find speed alone would be a game changer for current models. I hardly find any task anymore that the current frontier models can't do with max reasoning after several rounds of feedback (provided sufficient instruction and the right harness). But waiting an hour or more for reasoning to finish is getting really cumbersome. If they could do the same in seconds (and for cheap of course), I'm pretty sure we'd pretty soon see major software companies pop up that are run by a single human.

There’s some kind of tradeoff between speed, cost, and quality for every application. I would be perfectly happy with a model 6 months old that was 50x faster for many uses. Right now I use either Opus (for smart stuff) or Flash without thinking (for fast stuff). I would take an even dumber model for more speed (lower latency in particular).

Already models have gotten really good at a lot of things.

A lot of people would probably be happy to stick with the same model for a year or two if it’s 10x faster and cheaper.

And perhaps older models can become cheaper over time as newer models come out on new silicon for a higher price. That incentivizes people to stick with older models.

They expect a sort of breakpoint at which each subsequent model version will only be marginally better than the previous ones, thus allowing them to retain their value for some time. Their business doesn’t work if each year the new model demolishes the previous one in terms of performance.

pretty much everything is “1 or more versions behind” by the time it comes out. the question is whether or not it’s still useful? at some point, presumably not every application will need the latest cutting edge huge model.

You could take your silicon chip and have it re-etched only with model diffs for an upgraded version.