alt Hacker NewsIf the agent can write the tests its supposed to pass and we are worried that agent produces quite a lot of slop (which is why we are doing the tests), then what is the defense against test slop? Test the tests?
Replies
Filligree • today at 10:43 AM
Read the tests.
Doing a pass where you just ask the AI to sanity-check the existing tests (against rules like “test against the spec, not the implementation” can also help.
KptMarchewa • today at 10:42 AM
Existing tests.
Manual testing.
I work on user facing applications, and since the models do not have good taste, testing the UX is essential.
If you spot a bug, usually the model will attempt to reproduce it in a new test case that does cover the actual issue.