But that is exactly the promise that the heads of the AI labs are making. Sam Altman is repeatedly saying he wants to be able to ask it "go discover a new branch of physics".

Perhaps it's not how you use LLMs, but it is the promise of AI.

For the record, I make a distinction between LLMs (a current tool that we have today) and AI (a promise of some mystical all-powerful science-fiction entity).

There is nothing intelligent about what we have today, artificial or otherwise.