But there is a very good chance that is what intelligence is.
Nobody knows what they are saying either, the brain is just (some form) of a neural net that produces output which we claim as our own. In fact most people go their entire life without noticing this. The words I am typing right now are just as mysterious to me as the words that pop on screen when an LLM is outputting.
I feel confident enough to disregard duelists (people who believe in brain magic), that it only leaves a neural net architecture as the explanation for intelligence, and the only two tools that that neural net can have is deterministic and random processes. The same ingredients that all software/hardware has to work with.
Brains invented this language to express their inner thoughts, it is made to fit our thoughts, it is very different from what LLM does with it they don't start with our inner thoughts and learning to express those it just learns to repeat what brains have expressed.
> I feel confident enough to disregard duelists
I'm a dualist, but I promise no to duel you :) We might just have some elementary disagreements, then. I feel like I'm pretty confident in my position, but I do know most philosophers generally aren't dualists (though there's been a resurgence since Chalmers).
> the brain is just (some form) of a neural net that produces output
We have no idea how our brain functions, so I think claiming it's "like X" or "like Y" is reaching.