It's almost a certainty that LLMs have a "core" that will essentially never become obsolete, possibly even 80% to 90% of their parameters. The rules of English and other languages, core ideas in math and science, all of history, nearly all literature, etc. We don't really understand what's going on inside LLMs enough yet to make good use of this, but one day we will have "core logic" neural networks with stable weights burned into ASIC that are doing the heavy lifting, with more dynamic continually-tuned models manipulating the inputs and outputs into those core models. There are also likely stable expert models on topics that don't change much that we could already do this with.

Inference costs cannot keep falling forever, but they do still have a long way to go.