The goal feels underspecified. I'm not sure I fully understand the perceived gaps between frontier LLMs and AGI. I read Gwern's original article and it was written two years ago. Several statements it makes were true at the time but no longer are.

In particular I'm not sure the statements about models making bizarre mistakes still holds at this point. It's been a long time since I saw good models do something that seemed genuinely stupid or strange. In the rare cases it happens it can be easily explained by aspects of the algorithms e.g. the model can't erase an already committed token and has to always build on it. I think a big part of why hidden reasoning helps is it gives the model a place to draft an answer where mistaken tokens can be ignored, hence why they're full of "Wait, but..." style tokens.

This leaves the (rather vague) inability to generalize. Is that true? How would one benchmark this? My perception is that LLMs are now superhumanly intelligent in nearly all ways, with exceptions missing only for continual learning (with the latest memory/note taking features, even this is arguable), and perhaps some very vague inability to have "shower thoughts" and "innovate" via non-obvious connections between things. But I see no reason why that is fundamental and the progress in maths proofs suggests it's not.

In other words, the notion that we need to massively increase param count might have sounded good in 2024 but seems kinda weird and pointless in 2026. What's the expected outcome? Again and again what I hear from colleagues and experience myself is that we're not really intelligence constrained at this point. Smarter models aren't going to fundamentally change how we use them. The roadmap looks more like exploring the cost/benefit landscape to figure out where AI should be applied and when not. That meta-work is hard to automate because the landscape is covered in a fog of war with super sparse rewards, and good results tend to come from intuition, experience and having contrarian opinions that turn out to be right.