alt Hacker NewsI have to say, I am starting to hate this line of reasoning. Yes, LLMs move extremely fast and a lot of improvements are done in a short amount of time.
And there might be a point to these arguments, vaguely. However:
There never seems to be - any - kind of counter example or reasoning behind the rationale. You have an in depth and empirical study, done by researchers who, frankly, now their shit (most of the time)
And on the other hand a random internet comment saying "nope" because...the models aren't the latest.
If the latest models really would make a difference, you should at least provide some kind of evidence towards that. As it stands though, every time these comments come up this is missing.
There seems to just be a vaguely defined understanding that "everything changes all the time, and nothing you ever research is transferable to state-of-the-art models"
Which brings me to my second point about these kinds of arguments:
LLM models often - aren't - fundamentally different. Yes, they are vastly more capable. And yes, there are emergent properties. But at their core, they function very much similarly. And for quite a while now, there have not been any of these drastic changes we saw when LLMs first become "good enough" for agentic coding.
I am tired of dismissing empirical evidence and studies every. single. time for reasons without evidence and seemingly a vague sense of "no, but my model is different"
Replies
I sympathise completely but a ~3B parameter model and ~3T parameter model are going to exhibit very different behaviour, one can only infer so much large model behaviour from the former.
The models they are using are Nemotron3 series from last year. For reference at that time, the state of the art was Sonnet 4.6 and GPT was essentially unused for coding. Currently, you can run Qwen 27B in your gaming PC and it will absolutely trounce every model they show in their report.
This is a field that changes significantly every few months, so using 1 year old models essentially invalidates the entire report, as they are 2 model generations behind and the newer generation models are heavily RL-d around their harness. For example they put a lot of emphasis on context management but that kind of context management isnt done anymore because many models now have 250k or 1M context and inferencing the models heavily rewards cache hits so you should never touch the context until you decide to compact. They comment on plan mode, which is something that Claude decided to remove from their client because they dont consider it needed at all anymore.