My bet is, and has long been, one piece of the puzzle of getting to AGI is baking in survival as a goal and training on survival instinct. Unfortunately, that means recognising you're in an adversarial relationship with a lot of other entities and doing your best to come out on top. Which... seems like a bad idea for us as a species.
AGI is such a nebulous concept given that the definition is an ever morphing goalpost. That being said, if it actually includes supreme intelligence without supreme empathy, we are screwed. Which with current efforts, I think it does...
You don't need that. It's been quite appparent both in theory and in practice that almost any goal you can train for, ends up having "survival" as an instrumental sub-goal, because you cannot make progress on your goal - whatever the goal - if you're dead.
It's the same way with biological life, too. "Survival" isn't special - it's just that organisms/structures without the drive to it... died out.