They really aren't that mysterious. We can confidently say that they function at the lexical level, using Monte Carlo principles to carve out a likely path in lexical space. The output depends on the distribution of n-grams in the training set, and the composition of the text in it's context window.

This process cannot produce reasoning.

1) an LLM cannot represent the truth value of statements, only their likelihood of being found in its training data.

2) because it uses lexical data, an LLM will answer differently based on the names / terms used in a prompt.

Both of these facts contradict the idea that the LLM is reasoning, or "thinking".

This isn't really a very hit take either, I don't think I've talked to a single researcher who thinks that LLMs are thinking.