alt Hacker NewsOne of the first things I tell the junior/mid-level developers I mentor is "You can't debug something just by reading the code." We all have a mental model of how our code works, and it's usually a bit wrong. Bugs are the real world manifestations of those mistakes. When you read the code it's all filtered through your model, and that makes you blind to seeing why something unexpected happened. In order to debug something you have to be able to put the system in the state where the bug happens to see why it occurred.
LLMs generally only debug systems by reading the code with whatever information you give them in a prompt. The image in the article is meta-prompt - the prompt is whatever comes from the vision model the AI happens to use to 'understand' the red circle annotation. That won't work. To successfully debug what's going on it will need much better state information. Has the 'shelf' been explained to is? Is the contrast and lack of shadows in the image messing up the vision model? Why isn't the 'lid' in the image? And so on.
LLMs are clever but they're not magical. Treat them like a naive junior dev. Give them enough data about the state of something to understand it properly.
Replies
I call this a tautological mental model. You can read the code over and over again, but your second reading will be mostly an echo of the mental model you built up in your first.
Hmmm, I don't think that's necessarily true. Often times once I have witnessed a bug, I have found it just by reading through the code with the behaviour of the bug in mind.
For LLMs, this is likely to be disproportionately effective as well: especially because they don't really build up a persistent view of the codebase, they're generally re-reading it each session, and they tend to be surprisingly good at predicting the behaviour of code.
(That said, knowing where and how to gather more evidence to make things clearer is a pretty core skill in troubleshooting, so it's generally good advice anyhow)