When Modernization Breaks Working Code: A Study of LLMs on Legacy PHP

> We study a common intermediate setting: a developer pastes one complete file into a chat assistant [5], which sees the file’s internal structure but not its callers or dependencies. We evaluate the first answer, not an agent’s result after iterating on errors.

Very unsurprising results given this methodology. The agentic approach where models are allowed to iterate on changes is a very big part of what has made LLM coding tools actually useful.

The average age of the explored models is ~1.5 years. They discuss PHP 8.3, which was released in 2023 and has not been actively supported since 2025. Agentic coding isn't used.

I'm not sure what the point is of publishing results on the mid-2025 state of the art in October 2026.

Models:

- Claude Sonnet

- Gemini 2.5 Flash

- Gemini 2.5 Pro

- GPT-5 Codex

- Llama 3.3 70B

It's inconvenient how quickly things are improving, makes potentially-interesting results like this dubiously relevant

One year ago I was very skeptical about such attempts. Today, it would be my first choice as LLMs evolved a lot and the result of carefully using a LLM for this task can be very, very good.

When code output becomes cheap, you write 15 million lines of code per weekend, and of course everything gets deprecated pretty fast version 1? that was like 4 geological ages ago (1 month) that's legacy.

[flagged]

[flagged]

[flagged]