alt Hacker NewsIt doesn’t. Decode is by far the most dominant and expensive part of agentic tasks and once prefill hits a certain achievable speed decode dominates the time in ant agentic flows with reasoning. There is a reason input is so much cheaper :)
It doesn’t. Decode is by far the most dominant and expensive part of agentic tasks and once prefill hits a certain achievable speed decode dominates the time in ant agentic flows with reasoning. There is a reason input is so much cheaper :)
Isn’t input so much cheaper because it’s just less work? Prefill is usually 10-20x as fast as generation.
If you have 200k context, but only 200t/s prefill, then you’ll wait multiple minutes when continuing an average session from yesterday. Or compaction - might take 15 minutes.