alt Hacker NewsYou are right only in so far that it is more economical.
But it is not the MBA's view of winning, it's just one potential conclusion you could draw from the bitter lesson of Machine Learning. As long as the need for more intelligence outpaces the economics of using intelligence, you'll get bigger models. This idea that small, fine-tuned models can outperform bigger models capabilities wise is mostly misinformed. They are genuinely good at other metrics, but sadly more actually means better in ML-land (most of the time at least).
I don’t think there’s a fundamental reason that performance has to be monotonic in model size or even training FLOPs. At least I don’t think it’s been proved to be so, so I think “misinformed” is a bit premature and sort of makes GP’s point.
There’s evidence that model size and representational capacity are not exactly the same, and that scale is maybe more important for learning than it is for representation (past a point). Consider the early work from the current neural scaling paradigm. The Chinchilla scaling study shows that smaller models can match the performance of larger models by training longer.
To GP’s point, if everyone is exploiting the scaling lever, few resources are being allocated to finding more efficient training algorithms that could let us work with right-sized models instead of pulling the scaling lever as hard as we can afford to.
I’ll end with a dramatic example from my field of materials science (which admittedly might not strictly generalize to LLMs). A lot of the field is pursuing the model scaling strategy, and it’s still paying off. But [0] recently reported competitive accuracy with much smaller models that run faster and can address much larger problems. The model architecture is pretty much the same, but they use a different training strategy and really focus on data quality
0: https://arxiv.org/abs/2504.21286