Show HN: Gemini 4 Argon plays StarCraft

Wow this is interesting to me from another angle. A friend of mine created a benchmark to evaluate LLMs on 9x9 Go (using KataGo opponents as Elo anchors) [1]. The relative rankings of this (StarSkirmish Bench v0.2) benchmark (GPT-6 Astra > GPT-6.1 Sol > Gemini 4 Argon > Claude Opus 5.5 > Claude Fable 5.1 > Claude Sonnet 5.5) matches the order of his GoBench rankings exactly, even though they are different problems at surface level.

[1] https://rolandgao.com/blog/gobench/

Is there anything interesting to learn from the agent written bots? Or are they just inferior versions of the best bots because the time limit is so short? Is there any indication that they’re trained on the code from the human written bots (like, do they magically “guess” creating functionality that requires experimental feedback or domain experience?) How much are the bots on rails applying the same techniques every time?

I think those kinds of questions would be more interesting than the raw win/loss ratios.

I am not sure I like this benchmark, but mostly, because I am not sure what it is really evaluating. It is not an idle dismissal.. multiple factors seem to complicate what it is supposed to look at ( just top of my head ):

- code generation speed - logic analysis - understanding of the game platform and how use of C may influence it - considered approaches, tactics and strategies

Don't get me wrong. It is is neat idea. I am just not sure what it measures.

This kind of makes me think about making better bots for Counter Strike based on human top level players, which i think its a very undervalued category of software. You can even forget about about CS and name any FPS and you'll be hard pressed to find a game with seriously good bots.

So this benchmark has the LLMs writing a bit to play StarCraft. Somewhat similar there was a post here a while back that had the LLMs playing StarCraft directly - https://broodbench.com/ and https://news.ycombinator.com/item?id=49766966

They should benchmark it against Pluto. It's a artificial neutral network bot that can easily take games off the best broodwar players.

Would love to see an LLM written bot play Red Alert 2/Red Alert.

I think with the abilities of AI improving, playing games would be a good benchmark as long as they're not trained on them.

I assume lots of bot code is part of the training data? So it's basically a version of how will they can improve upon those solutions public out there?

Does anyone know if there was a project having LLMs play Cities Skylines and see how well it manages to design and build a city?

This is an interesting benchmark - I like that time is limited, as it often isn't in other benchmarks.

It would be interesting to see the reverse, how long does each model take to get >50% WR against bots of various tiers.

Oh my gosh! It feels so cool trying this that is played by LLMs. I am sooo eager to try argon 4 and build many robotic projects.

How do they avoid the LLMs simply working with pre-training data? Which apparently they do, since the human written bots win. But it's a mystery, is what it is.

The video shows GLM but it's not in any graphs. Are there more complete charts anywhere else? Ones that include the 45 LLMs?

If you want to see human written bots in action or compete in the bot ladder yourself, try https://aiarena.net (SC2)

It's beyond insane that Astra achieves above 50%. Are models allowed to "play against" the specific bot they're up against? What explains the fact that it drops to single-digits against strong bots?

What if we let the agents play directly instead of writing bots?

How do the best humans do against the LLMs?

Any thoughts on building this for Starcraft 2?

why protoss?

[flagged]