And now the other aspect of the trade off is token budget allocation.
The author seems to be in a situation where you can burn as many token as you want. I don't know if that's a general situation.
(Even if you don't care about the environment impact of your computations, there is a dollar bill associated, and _someone_ cares very much about that.)
I can foresee a situation where devs will have to decide on how they allocate a fixed token budget - and then, faced with the option of "burning tokens to add a new feature requested by a customer for tomorrow's demo" or "burning tokens to maybe make the app faster in some edge case", the trade off will look a lot like the ones organisations made with human dev time.
This assumes that tokens are not going to get dramatically cheaper. I can't predict the future, but I don't see a path to that (or, are local models, and "a TPU in every machine" going to make the question irrelevant?).
I can definitely see a path were tokens get massively more expensive (let's meet six months after anthropic's IPO and check :D)
alt Hacker News
And now the other aspect of the trade off is token budget allocation.
The author seems to be in a situation where you can burn as many token as you want. I don't know if that's a general situation.
(Even if you don't care about the environment impact of your computations, there is a dollar bill associated, and _someone_ cares very much about that.)
I can foresee a situation where devs will have to decide on how they allocate a fixed token budget - and then, faced with the option of "burning tokens to add a new feature requested by a customer for tomorrow's demo" or "burning tokens to maybe make the app faster in some edge case", the trade off will look a lot like the ones organisations made with human dev time.
This assumes that tokens are not going to get dramatically cheaper. I can't predict the future, but I don't see a path to that (or, are local models, and "a TPU in every machine" going to make the question irrelevant?).
I can definitely see a path were tokens get massively more expensive (let's meet six months after anthropic's IPO and check :D)