alt Hacker NewsMixture of Experts is already used by pretty much all modern LLMs to address exactly this phenomenon.
Hopefully, future models can be trained to be even more aware of external knowledge, accessible through web search / RAG / whatever it will be then, and might not need to internalize much knowledge at all.
> future models can be trained to be even more aware of external knowledge
Then you need longer contexts, which is proving to a much more stubborn problem than general knowledge compression.