logoalt Hacker News

zozbot234today at 12:10 PM0 repliesview on HN

But text generation is quadratic after the KV cache optimization. If every decode step now has to recompute KV cache including its latest and most expensive tokens (even with a quick, "draft" model) that's even worse.