There's a large gap between making up words and an actually native text distribution. LLMs have a clear pattern, clear tells, a "feel" in English, and it's normally even more pronounced in non-English languages.

Lots of bias towards English sentence structure, idioms, etiquette, etc.