When are we as humans creative outside our training data? It's very rare we actually discover something truly novel. This is often random, us stumbling onto it, brute force or purely by being at the right place at the right time.

On the other hand, until it's proven it'd likely be considered a hallucination. You need to test something before you can dismiss it. (They did burn witches for discoveries back in the day, deemed witchcraft). We also reduce randomness and pre-train to avoid overfitting.

Day to day human creative outputs as humans are actually less exciting when you think about it further, we build on pre-existing knowledge. No different to good prompt output with the right input. Humans are just more knowledgeable & smarter at the moment.