alt Hacker NewsYeah, I've experienced similar stuff. Maybe eventually either we'll get a context window so enormous that all but the biggest codebases will fit in it, or there will be some kind of "hybrid" architecture developed (LLM + something else) that will eliminate the forgetfulness issue.
I find the whole idea of context window inefficient. The model that knows more than anyone could, can’t hold a memory of a codebase? I know it’s a limitation of the transformer design, but I find it quite disappointing that most of the investment is being spent on optimizing inefficient technologies rather than rethinking about the design.