The simpler and I think correct conclusion is that the LLM simply does not reason in our sense of the word. It mimics the reasoning pattern and try to get it right but could not.
What do you make of human failures to reason then?
What do you make of human failures to reason then?