You have probably seen examples of LLMs doing the "mirror test", i.e. identifying themselves in screenshots and referring to the screenshot from the first person. That is a genuinely novel question as an "LLM mirror test" wasn't a concept that existed before about a year ago.

It's not exactly difficult to come up with a question that's so unusual the chance of it being in the training set is effectively zero.