alt Hacker NewsI don't know much about LLMs but does that mean you have N^2 computation with the context size since every token needs to track how it relates to every other token?
Replies
acedTrex • today at 8:08 PM
For full self attention yes
I don't know much about LLMs but does that mean you have N^2 computation with the context size since every token needs to track how it relates to every other token?
For full self attention yes
Yes, except no with the KV cache. Because tokens aren't modified by future tokens you can cache the meaning of previous tokens. This makes the total effort linear over the entire context (or constant per forward pass).