You can code your prompts to read and write an external world model on the side. This is what most people do who are seriously doing games with LLMs.