Isn’t this essentially what MoE partially solves with varying levels of accuracy?

> Always curious when someone will figure out how we can elide most of the data from an LLM (but retain the logical ability). I don't actually need an LLM to have a very big internal knowledge base to be useful, so long as it can invoke a search tool...

I think this can be achieved already. Take a base model and train only on source code. In fact, the very early Granite models from IBM were like that though it didn't support reasoning which limited its performance.

You can do it too. I don't know how much it will cost to train on just source code repos. $10K in total? Not sure.