This shows just how completely detached from reality this whole "takeoff" narrative is. It's utterly baffling that someone would consider it "controversial" that understanding the world requires *observing the world*.
The hallmark example of this is life extension. There's a not insignificant fraction of very powerful, very wealthy people who think that their machine god is going to read all of reddit and somehow cogitate its way to a cure for ageing. But how would we know if it works? Seriously, how else do we know if our AGI's life extension therapy is working besides just fucking waiting and seeing if people still die? Each iteration will take years (if not decades) just to test.
I think it’s still difficult to conceive of this branch of computer science as a natural science, where one observes the behaviour of non-understood things in certain conditions. Most people still think of computer science as successively building on top of first principles and theoretical axioms.
Last year went for a walk with a fairly known AI researcher, I was somewhat shocked that they didn't understand the difference between thoughts, feelings and emotions. This is what I find interesting about all these top someones in AI.
I presume the teams at the frontier labs are interdisciplinary (philosophy, psychology, biology, technology) - however that may be a poor assumption.