Probably good to establish canonical spec documents up front that it can use and maintain as an independent reference?
Doesn't need to be "up front", but you do need to curate the context.
AI can't read your mind, and it will read what you (or anyone else) wrote.
That doesn't solve the problem of it hallucinating more design details and putting them in comments. The solution to too much authoritative documentation isn't more authoritative documentation.