Sure, but the same must be true for these models as well. They must be doing some form of reasoning, even if it's not formally correct and it relies instead on all kind of half-assed heuristics which only work inside certain unexamined boundaries, which leads to wrong conclusions when those boundaries are being trespassed, etc. And even if it's not the reasoning they report when asked. But all this can't be that different to how humans reason most of the time either.

I suspect humans have other ways that help with error correction and guiding the reasoning effort, but that's another story.

There is explicit reasoning where we externalise steps, but it is slow and cumbersome. The point is that when we make a decision without externalising each step, and then afterward explain why and how we made that decision, there is no evidence that explanation is true, and a lot that it is a retrospective fiction.

Of course, but it is rare.