alt Hacker NewsGPT-5.6 Sol can beat Factorio on peaceful mode
Comments
If you haven't been keeping an eye on LLMs playing games, GPT 6.1 Sol can new beat games 100% vision-only (without harnesses that give the agent extra information about the game). Pokemon is no longer a difficult challenge for it, even with additional difficulty restrictions such as hard mode and level caps. Recently, GPT 6.1 Sol beat Final Fantasy VII and its bonus bosses vision-only surprisingly quickly.
I suspect Opus 5.5 can do even better but there aren't as many people testing it.
Watching AI play games is fun, but I can't wait for someone to crack having small LLMs in games to enrich them in useful and interesting ways.
I bet it helps that Factorio attracts exactly the kind of obsessive players that'll leave a digital trail of training data documented in precise detail
Bit like warthunder guys can't help but leak classified info
One limitation of these type of demos (and similar with Minecraft, etc) is that: 1. Putting into "peaceful" mode makes response time a non-factor 2. These models have read and have access to every guide and video ever published on the subject. Most humans, especially on first play, learn game systems and interactions from scratch, then develop novel strategies and plans and "hypothesis test" their way into beating the game.
This is still impressive compared to where we were a few years ago, and being able to synthesize all existing information about a game and then play it to completion in a semi-realistic setting is great.
At a quick glance someone might mistake this for "brilliant AI solves Factorio", when in reality, given a similar but different game where it wasn't gifted all the information or training data (e.g. massive videos and screenshots) it might fare terribly (as others have pointed out in comments).
Factorio is a much easier game without the bugs being a pain in the arse as you expand - i wonder how Sol would handle the bugs
It doesn't seem to understand the core mechanic of Factorio, which is putting outputs onto belts and routing them to other inputs. It does this in a few places, but for the most part everything is put into chests and moved around by the player.
It is weird and almost alien how AI designs factories. When Claude announced Fable 5 they also did a Factorio demo that was the same kind of designs and a lot of hand feeding and direct insertion. In some sense it proves to me that llms are not just copying and are actually creating a bespoke solution. But I honestly expected more from frontier models
Honestly, I'm more interested in the harness. Having an LLM gardener that could expand outposts to connect to the main network, or just pop in to observe if there's any major problem to report could be a cool project for long-running multiplayer bases.
I see it at https://github.com/ryanmadden/autorio and https://ryanmadden.net/claude-plays-factorio/, but I'd like to ask - any plans to support blueprints? Or push it further into Space Age or dare I say.. pyanodons?
No matter the answer: very nice!
I tried a few recently on Civ 6 Deity, and they all lose pretty quickly.
I don't have the budget to do so, but if I do, it would be interesting to put different models in the same game with the same prompt to observe the behaviors that different models have when told to problem solve.
This is a much better benchmark as a concept (without knowing the details of how the problem is framed to the agent) than some of these ridiculous things the frontier labs are reporting (vending machine simulator? come on).
Can you report more in depth at stumbling blocks? It's a surprisingly linear curve.
I'm going to try it on a local run.
I wonder if there is room for "recommendorio" where it gives you AI advice on your current factory.
Is the model pre-trained with any knowledge of the game?
Automating fun was not on the top of my priorities but sure.
is that spaghetti factory i am seeing? :O
interesting artifact of over investment and subsidies of AI and datacenters
Good to know it's good at something, I suppose? Sol seems like it always just wastes my time and writes terrible code.
[dead]
[dead]
> The rocket launch was only achieved through significant ‘manual’ effort on Sol’s part as it papered over its late-game failures to connect certain production systems by repeatedly ferrying supplies between areas of the factory.
Robots: they're just like us!