The models they are using are Nemotron3 series from last year. For reference at that time, the state of the art was Sonnet 4.6 and GPT was essentially unused for coding. Currently, you can run Qwen 27B in your gaming PC and it will absolutely trounce every model they show in their report.

This is a field that changes significantly every few months, so using 1 year old models essentially invalidates the entire report, as they are 2 model generations behind and the newer generation models are heavily RL-d around their harness. For example they put a lot of emphasis on context management but that kind of context management isnt done anymore because many models now have 250k or 1M context and inferencing the models heavily rewards cache hits so you should never touch the context until you decide to compact. They comment on plan mode, which is something that Claude decided to remove from their client because they dont consider it needed at all anymore.