It's language. Language itself is the thing that makes us smart in the unique way that we are among the other animals, and it weirdly turns out to be transferable to machines to at least some degree.
I think you’re right but I wanted to note that we are discovering that other animals also have language.
At least 50% of humans have no "inner voice" and are not thinking in the same way as you. Many animals like dolphins, dogs, rats, crows are also very intelligent yet appear to only have primitive language capabilities.
A lot of human intelligence is really societal rather than individual, based on knowledge transmitted down through generations by writing (the real enabler). If you take that away then what you are left with is something more like an isolated hunter-gather tribe.