Code gen is the absolute best case scenario for LLMs though: highly structured language, loads of training data, the ability to automatically error check the responses, etc. If they could mimic reasoning anywhere it would be on this problem.

I'm still not convinced they're thinking though because they faceplant on all sorts of other things that should be easy for something that is able to think.