alt Hacker NewsI've had it find bug, I asked it to make test to trigger the bug, and then it figured out it's not a bug. It will absolutely do wish fulfilment
Replies
csomar • today at 9:08 AM
It'll find a non-existent bug - fix it - figure out it broke a previously working thing - try to fix again - etc..
The "keep improving" the code base prompt have been tried and it never works. The LLM has no consciousness of where to stop and where to draw the lines of reasonableness.
Yeah when these models find a bug i like to ask it to write a test that will fail if the bug is real and pass when the bug is solved.
It’s not perfect but usually it works pretty well, and I’ve had the model come back to me with oh actually the test passed, the bug doesn’t work exist
As a bonus, you’ve now got a test that can detect that bug if it comes up again.