alt Hacker NewsYou know how everyone thinks agents are bad at the thing they're good at and good at the thing they're bad at? It turns out I must be bad at testing, because I thought they do a reasonable job.
Replies
ngruhn • today at 6:51 AM
I thought so too but it's probably because I never review the tests. I made an exception recently and found tons of
assert(CONSTANT_CONFIG == valueOfConfig)
assert(prompt.includes("repo url"))
I don't think these results show that.
I think they show that additional prompting for testing approaches have wide differences in error rate (worst has twice the error rate of the best) but actually no extra prompting is pretty fine and most custom prompts are are worse than no prompt.