This kind of possibility was the subject of some discussion on r/localllama the other day: https://www.reddit.com/r/LocalLLaMA/comments/1wxvt41/my_qwen...

(in that case it seems likely to just be an overfit to the training environment, imo)