> It looks like AI is finally branching out of math.
Is it really, though? It's still a heavily mathematical domain, rather than one involving empirical research, where AI still struggles.
I agree, but it is the kind of math done the physicists' way (which I found recently asserted that AI cannot do) and not the kind which is formalizable in Lean.
"It's not much, but it's honest work."
We used to joke in my physics undergrad that they didn’t need a math department because we were doing all the stuff they were doing but with actual applications. Then the mechanical engineers said the same thing about us!
In my (limited) experience it does not struggle with empirical research — at least in the procedural aspects thereof — as long as it has access to the right data.
It is very good at understanding research goals, identifying appropriate datasets to work with, doing data wrangling and data science, determining appropriate metrics, running simulations, analyzing results, writing up reports with charts and tables, and proposing follow ups or “high potential next steps” which show a surprising amount of judgment, and if instructed to, executing on those too.
I use Astra and Fable (who delegate to cheaper models for coding) and have them review each others work, and only spot check things when they are done calling each other out. I rarely find issues, and when I do, it’s usually a misunderstanding of intentions.
I’ve only explored a few areas though, including computer vision, vector search, and wireless signal processing, all in applications that, as far as I can tell, are genuinely novel (though of varying levels of novelty and economic value, as the models are fond of pointing out to me!)
I prefer being in the loop and setting the direction — because I do not have unlimited tokens, hah! —- and I regularly identify new directions they had not considered or had prematurely dismissed… so I am a bit more useful than just a financier of tokens! But I can totally see how they could grinding ahead by themselves and brute force useful results, especially if given very concrete goals like a theorem to prove.
I don’t think we’re at Recursive Self Improvement yet, but I don’t doubt for a minute that this is behind the insane pace of development at Anthropic and OpenAI.