An LLM works better the more disparate world knowledge it has, even if it's not immediately obvious why it would be relevant. The model finds a structure to the problem you give it in a largely language-agnostic way that benefits from training on every language (these things are direct descendants of Google Translate), and even non-programming knowledge - the structure of your task might resemble an ancient Chinese poem that influences the model's response, for example. That structure is considered a form of compression, as some fascinating and illuminating recent 3blue1brown videos get into - a common pattern in Haskell or FORTRAN and a situation described in an ancient Chinese poem may all compress to something quite similar to your task, thus when the model compresses the idea of your task it immediately draws from those ideas.

There are "experts" which do divide parts of the model that are found to activate together for specific tasks, so they can be processed in parallel to join the result at the end, but it's nowhere near the granularity of a SwiftUI expert and a python expert. The difference in those things is so trivial from an abstract point of view that it would make no sense. They would be 99% the same.

Distillations also come into this but I'm highly skeptical you could make one guaranteed to only know programming and only in one programming language (especially with as small a sample set as SwiftUI relative to something like C) without its efficacy being hobbled by tunnel vision. Reminiscent of the SpongeBob episode where he empties his mind of everything except fine dining and breathing, then can't remember his name and goes insane. Beyond the basic concepts of general coding and the trivia of syntax, getting anything done requires a large intersection of disparate world knowledge and the ability to apply it to new situations.

This is a fundamental misunderstanding of how LLMs work. You can’t really specialize a model. You specialize the harness. A well-trained general purpose LLM doesn’t need examples in its training data, it can write good code in a new language you invented yesterday with just a spec definition. And it will perform better than a small model trained on lots of examples of your invented language. The reason is because of the “universal geometry of embeddings”, i.e all human languages have the same underlying pattern structure, so any model that is very good in any language is good in all languages. Attempting to specialize a model for a particular purpose often decreases overall performance. Fine-tuning is just a hack to make dumb models more reliable on limited tasks but they become incapable of doing anything else. Unless you are building a factory assembly line where a model is literally doing the same thing over and over, you almost always want a general purpose model over a specialized one.

> My model doesn't need to know a single line of python.

If I had to guess, the weights necessary to encode "how to program" are much larger than the final step of "output python."

> I don't want general purpose models. They try to be everything to everyone.

I think the vast majority of people do want general purpose models. They want to be able to ask it any question, or ask it to perform any task, and for it to do a decent job at it.

I agree that it's really hard (maybe even impossible) to build something that's everything for everyone. But your average (or even above-average) LLM user doesn't want to choose from a catalog to stitch together a model that does just what they need.

I do think for certain domains this is useful and will make sense: the model backing a coding harness doesn't need to know about the politics of 400BCE Rome. But I'm skeptical that many software developers will want to do what you propose, picking knowledge bases that are tailored to their current task or project. And at any rate, for web-based chat interfaces, most users just want to type a query and get an answer.

> I want to click together a model that is laser-focused on what I am doing

This is roughly what multi-agent systems are built for.

This is possible with models too, but "making one on the fly" is much easier with agent coordination rather than model weights, since they all speak the same language.

There is an IBM Mainframe vs Google Distributed system division here. Like Seymour Cray said - two oxen or 1024 chickens.

Chickens are harder to harness, so a lot of my work is in sled-dog territory for agent harnesses & command structures.

Sounds like unix philosophy. Or like Neo downloading Kung-Fu. Good either way.

If that is better (and possible) it will be baked in the tools. But is a model that doesn't know python better or worse at swift is what I wonder.

(Inexpert ramblings follow)

Part of the problem of this is likely that the deep meanings of words you might use in chat to describe a business problem or task that you wish to see implemented are essentially inseparable from scenarios in which they are used.

Putting aside the bouba/kiki effect and anything like it, complex words only have meanings from usage. That usage is built on grammatical structures that also emerged only from usage.

(This is something I was taught as a sort of fact but I gather it was basically abbreviated Wittgenstein? … who I cannot claim to have studied)

So what you're looking for is a language model where fundamental word meanings are encoded without the weight of knowledge of where they come from. This is plainly difficult, because complex words are used by extension and analogy, and these days, many are neologisms or portmanteaus, even ephemerally — developed and discarded within a single context.

Reasoning about language itself to its full meaning is quite hard.

Like my favourite word of the moment: "obscurantist". You see that and you have a glimmer of what it might convey. But why do you? How much of that comes from explicit grammatical knowledge of suffixes, and how much from simple experience of using words like obscured, informant, attendant, dentist, artist?

So a language model might be able to deduce what "obscurantist" logically means when applied to a tract or to a person. But without lots of parameters covering its use, could it properly grasp that in some circles it would be pejorative to the point of being deeply offensive?

I think the best hope for your pluggable knowledge base idea is model delegation: strong reasoning models that know how to dictate to smaller specialist models and draw conclusions from their responses. I find myself wondering if there's any way that can be done the same way that, say, Gemma 4 12B's integrated vision encoder works — within shared weights, somehow, without them to speak in some intermediate language, like a partitioned brain. But I find it difficult to believe that is pluggable at all.

Sounds like MoE but more pluggable. Not sure if anyone is researching something like this. I still think your swift or GIS model will need basic reasoning and coding to work, so it's more like multiple smaller models which you can load as needed, e.g. sub-agents for GIS needs the GIS model.

Another approach would be to have basic coding and reasoning model and then load specification for language and libraries into context, it could work for self-hosted models, but I don't want whole specification of the language to be send to API and waste tokens on that.

This would defeat the AGI narrative/belief that so many building these models have

The problem with this idea is that knowing Python makes the model a better Swift programmer, as does a higher-number of parameters during training. So you'd be so much better off with a 90B general purpose model trained on everything anyway.

The LLM is the reasoning engine that uses natural language. You’re describing skills. It’s the natural evolution.

Admittedly I’m pretty ignorant of the details, but I thought this was the mixture of experts architecture

This is going to be how I write my resume now: how many billion parameters of knowledge I have on each topic

I would love this but I think the General Reasoning and Make No Mistakes modules would be massive.

> So if I'm e.g. coding a SwiftUI app for navigation, I'd take 9B of basic coding and reasoning, add 10B of swift/swiftUI, add 5B of GIS/geography knowledge and another 5B of frontend app design knowledge. My model doesn't need to know a single line of python.

Tell me you don’t know how llm work without telling me you don’t know how llm work. That’s not how they work!

I strongly suspect this will be the future

> So if I'm e.g. coding a SwiftUI app for navigation, I'd take 9B of basic coding and reasoning, add 10B of swift/swiftUI, add 5B of GIS/geography knowledge and another 5B of frontend app design knowledge.

Aren't you describing RAG or even MCP servers? Heck, nowadays you get that also with agent skills and specialized tool calling.

This is possible today with an agent such as OpenClaw and hermes