I have been playing with this. Having tiny models provide more contextual information when an NPC speaks is cool but the "small enough" models tend to go off the rails.
It's fun if you have a handful of people playing, but it would quickly go towards negative reviews if you tried to sell it.
Nothing great was ever built by someone who was sweating the reviews. Fuck 'em, do what feels right to you.
I don't think an LLM should ever be having a two-way conversation with the player. It would be more suited to 2 npcs having conversations amongst themselves. All dialogue delivered to the player should be human written.
At most it should be mildly tweaking existing dialogue, e.g updating a line to reflect game events. But as soon as you present a prompt window to the player the entire illusion that you are in a game world instantly dies.