This is intriguing because it reveals that LLM hype merchants are actually at least as afraid of AGI as critics; they too would rather not have AGI if the AI achieves it in ways they cannot scrutinise, when it is clear that for any AI to achieve a level of intelligence greater than humanity, scrutiny in human terms cannot really be generally possible.

The way we think is our limiting factor, and for the AI to outdo us, it is going to have to be unchained from that.

Wrestling with this is the future of AI, and reckoning with the fact that a superintelligence might operate in ways we can never understand, is crucial to keeping it from destroying us quite incidentally.

(FWIW I don't think intermediate traces in LLMs, or indeed even LLMs generally, are what will get us to this point, but my feelings are instinctive)