> what actually is happening inside an LLM has nothing to do with conscience or agency
What makes you think natural brains are doing something so different from LLMs?
For starters, natural brains have the innate ability to differentiate between things that it knows and things that it have no possibility of knowing...
Any amount of reading into how we understand brains and LLMs to work.
Structurally a transformer model is so unrelated to the shape of the brain there's no reason to think they'd have many similarities. It's also pretty well established that the brain doesn't do anything resembling wholesale SGD (which to spell it is evidence that it doesn't learn in the same way).