alt Hacker News> So if I'm e.g. coding a SwiftUI app for navigation, I'd take 9B of basic coding and reasoning, add 10B of swift/swiftUI, add 5B of GIS/geography knowledge and another 5B of frontend app design knowledge.
Aren't you describing RAG or even MCP servers? Heck, nowadays you get that also with agent skills and specialized tool calling.
Replies
I don’t think so because those both live in the context window and as such pollute it when they’re not performing optimally.
I think having unused or rarely used weights doesn’t influence the results as poorly as RAG injecting irrelevant facts.
It sounds to me like some sort of “dynamic MoE” where you can add/create or remove experts on the fly.
I think what you’re describing is the closest approximation we reasonably have right now though.
I don't think so?
Definitely not MCP, as that pulls info into the context. Unless contexts become REALLY big so that I can add 10B in swift knowledge, that's not gonna help me.
Possible RAG? I don't know enough about how that works, but I think that's not quite it either. I don't want to import facts like "the swift standard library contains a reverse array function", i more want to import knowledge - e.g. the parameters used to generate the text to reverse an array in swift.
Tool calling wouldn't do it either. You'd have to encode every single possible bit of useful info into the tool call, and the tool response would have to encode every piece as well (variable names, function scopes, types defined in other files, etc). E.g. how does it find a bug, if you have to pass understanding back and forth between the brain that understands debugging and the brain that understands THIS code?