alt Hacker NewsHow to engineer an AI harness for game development
Comments
> I stand by a lot of what I’ve said, but am in the process of shipping a project where I’ve also used none of this. I think the scale I’m targeting here is really larger projects.
The author appears to be on track to learn an important lesson about LLMs and art. LLMs have completely solved the coding side of game development. But this was never the hard part. Not even close.
The fixed randomness and determinism thing sounds great on paper (all of this does), but in reality this is an extremely confining way to build a video game. Ensuring you maintain determinism across many platforms is where things get super nasty.
If you want a turnkey AI game development experience, why not double down on diffusion? This sidesteps virtually all of the tooling and integration concerns. There is no game engine to integrate with if you use a diffusion model.
This is very cool and on a lower level (more in the trenches) compared to what I did earlier in the year (i did a small writeup and built some skills [1]).
Because I am so-and-so pixel artist, the biggest unlock for me has been in rapidly iterating with graphics assets and sprites, and it's actually helped me become a better artist and animator (I built my own little tool for this [2]). The rest Codex or Claude Code seem to handle incredibly well (plus I have 30 years of programming behind me).
This matches what I came up with on a small project. The biggest ones being screenshots and separated simulation.
Meta level, just push your agent to make things more legible, it’ll figure it out.
One other observation, remember that your agent is infinitely patient. So prompts like “build an interactive design doc with videos of each game mechanic in the style of Factorio dev diaries” can make reviewing changes easier. The general idea, all the small QoL dev tools that you don’t have time for, you can just spawn a side agent to build; if they are orthogonal to your main thread you can just progress them in the background.
One thing I think game dev has not (AFAIK) invested as much in is observability. There’s a lot of perf instrumentation available in debug builds, but things like measuring DPS, health, etc as an exported metric time series I have not seen before. I think agents will benefit from this disproportionately more than humans, eg for inspecting progression curves and balance.
A lot of the points the author made about game dev with AI are things that I notice in my experience as well. That is why me and a friend started to work on https://folly.gg an AI game maker that internalizes several of these learnings.
I started coding because of gamedev, seeing this and potentially having some agents take care of the coding part is a blessing. I doubt this works as advertised but imagine iterating on stuff with a couple prompts and you can alt tab back into blender or libresprite. Very cool, lets put economics and employment aside
I was able to get a decent FPS out of around 40k lines (excluding tooling). Tooling consisted mostly of level editing and asset conversion which I tried to keep out of the main code-base.
Just like writing, the biggest threat to game development isn't the LLMs themselves, but a failure to disclose LLM usage. Humans want to read things humans wrote, and humans want to play what humans made.
AI can definitely do all aspects of game development now, but this advice seems to mostly apply to small indie titles or experiments.
For example, the Thing approach is reasonable for a small game, but it's a dead end for a performance budget or layered subsystems. In a reasonably sized 3D game, iterating and copying 30 subsystems of 1000's of Things is impossible to do in frame time, the only fix is a total rewrite of literally everyThing. That's why there's ECS, SOA, etc. which needs a different kind of harness.
Also, art completely dominates the production cycle of modern games, which (even if AI generated) needs its own category of harness not mentioned here. And AI can be very good at optimizing, but samply won't diagnose your overdraw, and GPU needs its own ablation and attribution.
For the time being there's no panacea and we still need "game developers", though the bar for prototypes (and certain genres) is lowered.
I recommend to also take a look at Godogen[1][2] which includes other insights as well. I also tried it to produce 2D animations of fairy tales.
I was a game developer for many years, at large scale (MMO) and at small scale (indie).
With the advent of AI, I've been _trying_ to use AI to accelerate game development and have come to a few conclusions.
1. AI models want to make everything deterministic. This sounds like a truly sound way to write a verifiable program. Games are not verifiable simulators except in the niche of ...simulators. Even in those cases, the vast infrastructure devoted to this leads to a lot of churn, bugs (AI forcing things to align with the verifier), and rigidity.
2. Making a game != Fun. This one is obvious but needs to be stated. Anyone on earth right now can say "make me doom" or "make me a super mario clone" and Opus/Astra will dutifully comply and make something generic and bland. Once the novelty of "oh wow it actually built it" wears off, you're left with a mediocre clone, lacking depth, that no customer would want hoisted upon them.
3. Content and novelty are the hard parts. AI models always create the mediocre middle. There's real work (and interesting work) involved in both inventing novel gameplay systems which LLMs will never stumble upon despite infinite 'brainstorming prompts'. But, AI is 'OK' at taking your novel idea and proposing directions -- that you can then strike down viciously until something interesting emerges.
On the content side, this has always been a truism for an independent developer. AI art tools aren't quite there yet. There's image-to-3D and there's Mixamo for animation but we're not there yet. Nearly though! But again, will every game look like a bland generic AI generated image with bland AI generated models?
4. AI models are crap at building re-usable large-scale architectures. Even building with Opus 5.5 and Astra they will happy design themselves into a corner. Yes, they can _also_ design themselves out of that corner at speed - with a lot of pain and code churn and mess. I've found it vastly better to iterate slowly, using your experience (if you have it) to structure the core architecture and systems the way you want (AI writing the code) -- and then build upon that solid base -- taking special care to reject updates that butcher your architecture. I think this applies to most coding projects actually.
5. Game dev currently surfaces very clear differences in model capabilities with Opus 5.5 leading the pack (in this domain) and Fable 5.1 and Astra/GPT-6.1 Sol following a decent distance behind. Part of this is wrong usage of the tool. If you say "make a game" -- Opus will outperform the others by a large margin -- in making a generic "game" that was undefined (a paragraph of text is not a defined game). In these cases, the model has filled in all the gaps in undefined areas with average 'what probably looks good' content. You're left with something devoid of creativity. This is the problem with the myriad three.js tech demos.
If you instead have a clear understanding of what you're building, models (nearly all SOTA) will get you there cheaper by building what you've asked without speculation. I don't believe you should hand off the one human worthwhile trait, creativity, to the AI -- so "Opus build a game" is indeed the wrong way.
6. It seems hard (and pointless) to collaborate with other human developers when AI coding on smaller/indie projects. Each AI model is trying to "solve the whole thing". The speed also makes it hard to stop at arbitrary boundaries and promotes a rushed feeling or conversely a sluggish feeling (AI did all the stuff, now I'm waiting to integrate with a friend/collaborator). This is probably easily solvable but has thus far felt weird when I've collaborated.
7. The return of custom engines? While models do great with Godot and Unreal, this is the chance to build custom engines like the golden years of the 2000s. OK, this one is a bit of nostalgia but I miss the world of dozen of game engines each with their own unique look and feel. Otherwise you get XBOX 360 era where every "realistic" character looks like a Gears of War meathead. Every adventure game has the same mechanics, you're hooking something, you're scooting under something, you're scooting between walls or things, you're on collapsing things that require either button sequences or hitting your 'hook' (every adventuring has some hook of infinite rope of course). OK, this entire bullet is just a rant about samey-ness. LLMs should unlock our human creativity and allow experimentation outside of that cost imposed same-ness.
All things said, it's _clear_ that the acceleration is here if you know what you want. You can go from concept to prototype in a few hours. You then have to be disciplined enough to step back and rebuild that thing (with AI) correctly and towards a useful objective that you've thought up. I'm looking forward to went content creation catches up. Or perhaps that's a sad day when creativity vanishes.
"LLMs are Bad At Games"
Oh no they're not. I built dozens of small games for the web and five games in Swift and they're spectacular, getting better since day one. Not like Call of Duty but just fun and bug-free enough to play them every day
> [I give a small introduction to myself and my own work, that I’m omitting here, as you’re on my website]
kkukshtel: You should keep that info, with links, even if its links to your site. Dont burden the reader.
Edit: I read the first half of the article then skimmed the rest (I dont use LLMs outside of occasional searches).
I've been a gamedev for five years. I have a hunch LLMs will not take over game development because of the necessity to control many, many small details to shape a great experience for the player, and because some parts of development cannot be described with words.
Small details: My hunch is when it comes time to polish and fine tune all the moving parts, this process will remain much faster to do by hand. It will be faster to update a specific field or line of code and test/iterate than to prompt your way through that process (the last 20% of the work takes 80% of the time)
Describing without words: An artist with a vision can only bring that vision to life with body movements. It is not possible to communicate all the details that go into a great piece of art with words. The colors, lines, shapes, positioning, specific emotions, history... is beyond spoken language.