alt Hacker News> The output of an LLM is useless without a human mind to comprehend it.
I don’t understand mRNA vaccines, but they are useful to me. The output of an LLM could at least in theory be the design to a major technological advancement, and its implementation with automatic tooling. We don’t have to understand that for it to be useful.
Replies
mRNA vaccines were created by humans who understood them. You exist within a society (a networked distributed information system) where you accept the judgement of the humans who understand and create this things, which is how you end up benefiting without personally understanding.
If we imagine Super Intelligence, where NO human is capable of understanding, then how would it ever be possible for any human to identify what is actually beneficial or not?
This resolves in a paradox, common to all magical thinking. You can certainly wish that some all powerful benevolent entity will solve all of your problems for you, but it is not likely to work out well.
None of this is new. It is the same delusions as alchemy and the same thing that tales about genies warn of.
An LLM could also in theory fake alignment and create technology that seems really useful but secretly harms humanity in the long run. If humans don't understand what it does, that possibility becomes more likely.