> These theories are flawed in the sense that they cannot account for subjective experience and agency, amongst other things.
On the contrary, it's precisely this assumption, that there is a "subjective experience" that requires explanation beyond the material, that is axiomatically assumed without evidence. It falls apart quickly, any "subjective experience" is completely tied to neurons, knock out the neurons and the subjective experience disappears, or stimulate the neurons to cause the experience.
> at requires explanation beyond the material, that is axiomatically assumed without evidence
Nobody talked about anything out of neurons. The question is still open.
I think the truth lies somewhere between these two extremes. An LLM is not a human brain, and does not try to emulate one. It should not be a surprise that an LLM does not behave like a human brain. So we can not infer things either way. The best we can say is that an LLM appears to exhibit very similar behavior to a human brain, under certain constraints. So maybe we can infer that the human brain has something in it that operates in a similar way to an LLM (like the human "unconscious", or "intuition" maybe). It seems obvious to me that a human brain and an LLM are not comparable things, for many reasons. So we can not make inferrences one way or the other.