alt Hacker NewsThat's not correct. Even a toy like an exponential weighted moving averaging produces unbounded context (of diminishing influence).
That's not correct. Even a toy like an exponential weighted moving averaging produces unbounded context (of diminishing influence).
What do you mean? I can only input k tokens into my LLM to calculate the probs. That is the definition of my state. In the exact way that N-gram LMs use N tokens, but instead of using ML models, they calculate the probabilities based on observed frequencies. There is no unbounded context anywhere.