How long before they will raise the amount of context it can hold?
Or, it there a ceiling that we can't go passed?
The context length scales quadratically in terms of compute, so barring algorithmic improvements, there's definitely a limit.
alt Hacker News
The context length scales quadratically in terms of compute, so barring algorithmic improvements, there's definitely a limit.