alt Hacker NewsBenchmarks:
| Benchmark | DS-V4-Pro | DS-V4-Flash | DS-V4-Pro | DS-V4-Flash | GLM-5.2 | Kimi-K3 | Opus-4.8 | Fable 5 |
| | 0813 | 0731 | Preview | Preview | | | | (w/ fallback) |
|--------------------------|-----------|-------------|-----------|-------------|-----------|-----------|-----------|---------------|
| HLE (wo/w tools) | 42.7/60.0 | 37.8/51.5 | 37.7/48.2 | 34.8/45.1 | 40.5/54.7 | 43.5/56.0 | 49.8/57.9 | 53.3/63.0 |
| Terminal Bench 2.1 | 87.9 | 82.7 | 72.1 | 61.8 | 81.0 | 88.3 | 85.0 | 88.0 |
| NL2Repo | 61.5 | 54.2 | 38.5 | 39.4 | 48.9 | - | 69.7 | - |
| Cybergym | 83.3 | 76.7 | 52.7 | 38.7 | - | 80.0 | 78.3 | 83.1 |
| DeepSWE | 62.7 | 54.4 | 12.8 | 7.3 | 46.2 | 67.5 | 58.0 | 70.0 |
| Toolathlon-Verified | 74.1 | 70.3 | 55.9 | 49.7 | 59.9 | 76.5 | 76.2 | 77.9 |
| Agents' Last Exam | 25.7 | 25.2 | 16.5 | 15.8 | 23.8 | 27.6 | 25.7 | - |
| AutomationBench (Public) | 31.8 | 25.1 | 12.8 | 10.8 | 12.9 | 30.8 | 27.2 | 29.1 |
| DSBench-FullStack | 71.1 | 68.7 | 41.8 | 37.0 | 61.8 | 73.7 | 71.6 | 77.2 |
| DSBench-Hard | 67.2 | 59.6 | 31.1 | 25.8 | 54.5 | 63.0 | 71.7 | 68.3 |
Replies
The timing looks like they are trying to take the wind out of Qwen's sails by releasing this on the same day that Qwen released the weights of Qwen3.8-max. Or maybe it's coincidence...
For comparison I looked at Qwen's claimed benchmarks for Qwen3.8-max (https://qwen.ai/blog?id=qwen3.8). Assuming each published set of benchmarks is believable, it looks like v4 Pro 0813 is better on average but overall performance is comparable. Pro 0813 is much cheaper. If you don't need vision capabilities then you don't have much reason to use Qwen3.8-max.
- 43.6 on HLE (Presumably without tools). Pro 0813 is a little worse.
- 86.6 on Terminal Bench 2.1. Pro 0813 is better.
- 55.9 on NL2Repo. Pro 0813 is better.
- 27 on Agent's Last Exam. Pro 0813 is a little worse.
- 72.5 on Toolathon-Verified. Pro 0813 is better.
- 56.6 on DeepSWE 1.1. If the DeepSWE listed for Pro 0813 is the same version, then Pro is better.
- 27.3 on AutomationBench. If the AutomationBench (Public) listed for Pro 0813 is the same, then Pro is better.
I guess we do need to wait to see if the upcoming DS pricing increase is enough to change the value proposition. As it is now, they could double or triple prices and it still would be a better value to use DS. I bet they know that.
Geometric mean of all these benchmarks :
* GPT-5.6 Sol: 65.5
* Fable 5 (w/ fallback): 64.5
* Opus 5: 64.0
* DS-V4-Pro 0813: 62.5
* Kimi-K3: 62.3
* DS-V4-Flash 0731: 55.8
* GLM-5.2: 47.3
So it's a Fable class LLM?
DSV4Pro vs Fable5
HLE w tools 60.0 vs 63.0
Terminal Bench 2.1 87.9 vs 88.0
Cybergym 83.3 vs 83.1
DeepSWE 62.7 vs 70.0
Toolathlon-Verified 74.1 vs 77.9
AutomationBench (Public) 31.8 vs 29.1
DSBench-FullStack 71.1 vs 77.2
DSBench-Hard 67.2 vs 68.3In classic reddit fashion the post you linked to is now deleted
IMO the HLE scores without tools seem to align better with real world performance of the models.
To me it feels like the difference between "RL performance" and the pretraining / base "knowledge".
Yes you can RL terminal bench to the moon but does the model hold up on out of distribution tasks?
Kind of like trying to navigate a dark room with a laser light, vs a flashlight. Laser is going to go a lot farther much more efficiently but only if you are already pointing it at the right place.