It seems pretty clear to me though that being good at intellectual tasks / the sort of usefulness we ascribe to LLMs doesn't strongly correlate with awareness.

Even just within humans - many of the least intellectually capable humans seem to have a richer supply of the traits associated with awareness/being than some of the allegedly highest-functioning.

On average you're far more likely to get a sincere hug from someone with Down's syndrome than from a multi-millionaire.

But I'm more interested in this when it comes to the animal kingdom, because while ChatGPT is certainly more useful than my cat, I'm also pretty certain that it's a lot less aware. Meaningful awareness - feelings - seems to be an evolutionary adaptation possessed by k-strategy reproducing vertebrates. Having a small number of kids and being biologically wired to care for them has huge implications for your motivation as an animal, and it's reasonable to think that a lot of our higher emotions are built on hardware originally evolved for that purpose.

(Albeit the evolutionary origins of that are somewhat murky - to what extent mammals/birds reuse capabilities that were developed by a much earlier common ancestor, or whether it's entirely parallel evolution, isn't known afaik - but birds seem to exhibit a similar set of emotional states to mammals, that much is true).