Yes LLMs can play chess and yes they can model it fine

https://arxiv.org/pdf/2403.15498v2