alt Hacker NewsCached.
they are the first part of the input and it contains no user dependent variables, so the model is in a known state that it can reuse across all users, it does not need to recompute all that inference
Cached.
they are the first part of the input and it contains no user dependent variables, so the model is in a known state that it can reuse across all users, it does not need to recompute all that inference
Unless they are using a linear architecture, the compute cost still scales O(n²) for n tokens, and nemory cost scales O(n).