Not sure about LLMs but I yearn for someone to train neural nets to play the NPCs in a shooter game well but without having aimbots or seeing through walls - basically a “true” version of F.E.A.R. AI where they play simulated squads that overwhelm the player with various tactics and adapt to their playstyle. Or even just make increasingly complex state machines and goal oriented systems which might be more performant but still encode the main rules.

This... Hopefully inference can become so cheap games can have simple LLM's integrated into them. Imagine an RPG with LLM's and with rich worldbuilding; like Fallout NV with LLM's. Honestly, scary!

I have been playing with this. Having tiny models provide more contextual information when an NPC speaks is cool but the "small enough" models tend to go off the rails.

It's fun if you have a handful of people playing, but it would quickly go towards negative reviews if you tried to sell it.

It's a funny situation at the moment because most attempts are just sort of shoehorning LLM chat into a game that doesn't really need it, and then gamers quite rightly mock the results which don't really add anything to the game.

I think there's a bit of a lack of knowledge of game history on both sides, because we already know it can be fun when a game is designed around it. It worked in Façade (2005!) and it worked in Starship Titanic (1998!!), and both of those could have worked even better with the technology we have today.

One type of game that seems like an obvious one to me, is something a bit like The Sims except that the characters' decisions and speech are LLM-controlled. Maybe one of the characters is player-controlled, so you can interact and talk with everyone else. Or to describe it from the other direction, it'd be a kind of interactive fiction like AI Dungeon invented, but grounded in a physically simulated setting.

There has to be a AAAA studio building the new, greatest MMO, that also utilizes LLMs as part of the game loop, surely?

(Not sure how prohibitive it would be to do anything interesting with local LLMs of any kind in a big game, given how many platforms/setups you have to make the stuff run smoothly on, and the sheer amount of RAM they would bind)

With the aid of SOTA LLMs, you can encode specialist intelligence into computer programs without going through the roundabout route of training small LLMs. And even when you do go the route of ML, it doesn't necessarily have to be an LLM or even a Neural network.

I cannot wait for NPCs having a casual conversation. Doesn't seem too hard, as current chatbots are already role-playing.

I hope to experiment with using classic tabular reinforcement learning in game AI one day. Imagine a framework that makes setting up small Q-learning tables easy, and you can just sprinkle dozens of Q-learning algorithms throughout your traditional AI scripting.

These old basic algorithms often have mathematical proofs that they will converge to optimal behavior, and they do so fairly quickly. Turning the complex world of the game into a Q-table would require some creativity and judgement, but is not that hard I think.

In the end we would have a controllable AI that actually does adapt. That's my hope anyway.

Fable seems to be a potential franchise to do all this; imagine a "living fantasy world" where everything truly unfolds based on your decisions, giving NPCs all the functions/actions that a player has.

i think hooking in an llm to dwarf fortress for turning the very mechanistic descriptions into prose. same with the adventure mode talk. instead of a constant "well weve got beasts" an llm could read a bunch of the language tree to make a unique introduction for each.

same with the df artworks. get kitfox to design some art styles for dwarves, elves, goblins, and humans, and then instead of just reading

"this is a fine engraving of a dwarf. it concerns the events of 102 when ulrist fastbuttons was elected mayor. ulrist fastbuttons is sobbing"

SOTA LLMs would probably be fun opponents in turn based 4x.

I would love the revival of the RTS genre with better computer enemies.

SS13 but the ship's AI is actually an AI instead of a player in disguise

impossible, unfortunately, because LLMs suck at creative writing. without handholding, they are incapable of producing anything but cliche-ridden purple prose for young adults. you can feed it the first half of a book with lots of sex, violence, and profanity, ask it to continue the story, and it will degrade into safe slop within two chapters.

both American and Chinese labs aim their models at coding and knowledge work, so they penalize adult themes in post-training.

There are already AI-based mods for games like Grand Theft Auto V that make NPCs talk back and follow your verbal commands: https://www.youtube.com/watch?v=3_Eng-F4akI&t=106s

A little unpolished because it was bolted on by fans and not the original game developers, but otherwise very impressive.