Since programming is increasingly offloaded to LLMs and English is the main way engineers interact with code, it's interesting to see how LLMs reason in different programming languages.

In our benchmarking, we've found LLMs perform comparably between languages for one-shot coding submissions, slightly favoring more popular languages. But if you give frontier LLMs a harness and let them iterate / fix compilation errors, they actually significantly outperform in Rust. Meaning, they come up with more insightful ideas when developing Rust, than for example Javascript.

Scroll down to the language comparison chart: https://gertlabs.com/?agentic=agentic