There is a paper [0] that shows that LLMs, in fact, cannot write and modify ‘whatever’ pretty well.

[0] A Survey on Large Language Models for Code Generation: https://dl.acm.org/doi/epdf/10.1145/3747588