alt Hacker NewsI would love to see any good research projects about it but i have the feeling that Frontier with MoE is making too fast of a progress so that a customized model would always be worse and that the MoE part is actually going somehow in this direction.
On the other hand, at the GTC was a talk about coding in different lanugage (like spanish) and explaining that the quality between spanish and english is relevant different.
But i have not found a good article about the impact of learning data with practical experiments or even if the order of the learning data matters.
At least I think i remember that Meta mentioned having better and less data can be better than more data with lower quality.
As long as these models can explain to you facts about any other topics, its still overfitted for the task though.
Capability in LLM's is distributed throughout the manifold in subspaces. Even worse, the subspaces exist in superposition.
That is to say, there is no single 'python' part of the model. The python bit is spread throughout the entire model and overlaps with other pieces that have similar, but unrelated, capabilities. For example the python subpspace might be partially in superposition with cupcake recipes, Esperanto, and calculus. We need calculus in a coding agent but not the other two. However, separating them cleanly is almost impossible, and even identifying them is tough.
Internally the manifolds are highly inefficient and nothing like you would imagine something humans built would be designed. It's more like something that evolved in nature.