I know it’s a joke, but it does make me wonder if LLMs would even be good at assessing if an idea is “novel”.

I only have a rudimentary understanding of how neural networks work, but I wonder if rather than “understanding” what “novel” really means to humans, an LLM would be most likely to agree that something was novel based on having seen that specifically referred to as novel in its training data.

So that if you give it an example of something that already exists, but which was very recently invented at the point in time when the LLM was trained, and you ask “is this a novel idea?” that because it had several sources in its training data describing that idea as novel, it would say “yes that’s a novel idea”. Whereas what we really meant was to ask it if someone else had already thought of this thing prior to us right now in this later moment.

And then on the other hand, even if something was “novel” at the point in time when the LLM was trained, perhaps we would fare better to ask it “has anyone thought of this?” rather than asking if the idea is “novel”? And that even though it considers the idea novel in a way it would also be able to say that yes this has already been thought of.