This is prescriptive vs descriptive grammar, where descriptive grammar describes how people actually use language. Using "they" in that context is only wrong because someone or group decided that must be the rule.

In Swedish -

> Vi går till mitt hus

"Let's go to my house" - this is correct but not really said

> Vi går hem till mig

"Let's go to my place" - incorrect but the actual way to say it