One must be careful about what is meant by "language" here, which is actually language expression. LLMs don't possess conceptual content. They only operate on language expression.

Language as an internal conceptualization is still (obviously) essential to logical reasoning. The question of how language and logic relate can, therefore, concern either the internal language of concepts or the expressed language and its grammar, the latter of which is related to logic in that it reflects to some degree the internal language, but it is not identical with it.