https://broodbench.com/

TLDR they suck and many default to doing probe rushes.

If you are talking about take a screen shot, analyze move the window a bit, take another screenshot, move the window, repeat until they have seen enough of the map to make a decision, then decide what to click on, issue order.... this could absolutely be done. It's very similar to how LLM's have driven a car on a safe track.

But right now the response time would be so slow that the bot would not be close to competitive except with the most low level players. Especially because its hard to click on things that are moving at the same time as well.

Sometime soon, I'd like to see an LLM be competitive using only computer vision, mouse movement, and locked into a human possible number of actions per minute.

Even the most powerful frontier models on SOTA hardware are not fast enough for starcraft, which generally involves multiple informed decisions per second at peak moments.

You can train a specialized model for this, but a frontier model is ill-fitting.