alt Hacker NewsI am not sure I like this benchmark, but mostly, because I am not sure what it is really evaluating. It is not an idle dismissal.. multiple factors seem to complicate what it is supposed to look at ( just top of my head ):
- code generation speed - logic analysis - understanding of the game platform and how use of C may influence it - considered approaches, tactics and strategies
Don't get me wrong. It is is neat idea. I am just not sure what it measures.
Replies
NitpickLawyer • today at 4:13 PM
> I am just not sure what it measures.
All of the above, to the extent that they can be measured from a simple to verify end condition (stats, win rates, etc). Adaptability, ability to understand and progress on a task, and so on.
➕ show 1 reply
luckydata • today at 4:07 PM
general "agenticness"
Yeah giving a 1 hour limit for code generation makes the whole thing a bit incoherent. Plus, performance is just going to depend on how closely the models have memorized the best open-source Starcraft bots and can reproduce them.