alt Hacker NewsI find this - or perhaps the title - a bit surprising.
I've benchmarked GLM 5.3 and DSv4.1-F on my fully-annotated decomp of the Nintendo 3DS's kernel, which I have a good mental understanding of, tasking them to find vulns and other bugs (in Max mode w/ subagents). GLM 5.3 founds almost all the vulns in 30min for $22, while DS only found one vuln for $2 in 40min.
Perhaps DS works better where targets have low-hanging fruits than can be found fast?
Replies
Makes sense, GLM 5.1/5.2/5.3 is a lot better than DS v4.1, I found the same results in other domains.
Given how fast and cheap DS is, it's just an ideal model with enough "IQ" to let it loose. Another thing they left out of the article, DS becomes really good if you provide custom tools for the task, on it's own it's mediocre.
If you are talking about publicly known vulns, it's a bit moot since they should be in the training sets. If not, you just burned the vulns to that inference provider's training data (and any intermediary), and future benchmarks will be meaningless.
In your example, couldn't you parallelize DS's work more ? You could have 11 times as many agents for the same price.
A little off-topic: where does one use those models such as GLM or DS for this kind of reverse engineering tasks? I think I read many of them refuse to help with tasks like those on their official platforms.
Which harness were you using?
I think it really depends on what the data DS was fine tuned on. If your use case is very specific, it wouldn’t have distilled that knowledge well.
you can run 10 ds-f agents at the same time for the same cost exploring a much wider surface area.
What I find surprising is that DS Flash can do it at all.
I love DS flash, it is an amazing workhorse to implement plans created by more robust models (such as GLM). But a more fair comparison would be of DS Flash with GLM Flash.
Thing is that GLM 5.3 is many multiples the cost to run, and slower.
I have good results with DS4.1 flash because I can iterate faster. I either provide it with correction, or it discovers its failures via the harness. And seems to respond well to empirical evidence rather than go in circles.
So it might need some prodding, but it's likely in this case it was able to brute force after several runs and collecting some evidence.
Also depends which glm 5.3 you use. I used open router and sometimes had good, then appalling performance. It turned out the provider being routed to - some were fp8 and some were fp4. The fp4 ones glitched on anything complex. It is such a difference that I only allow fp8 providers in my allow list.