alt Hacker News> I recently deleted a whole bunch of automated tests because if the AI is going to write most of the code then I should test it to make sure it's good!
??
You say you deleted the tests, because you "should test it"? The logic seems inconsistent.
Sanity checking LLM-generated code with LLM-generated automated tests is low-cost and high-yield because LLMs are really good at writing tests.
Replies
otabdeveloper4 • yesterday at 4:57 PM
> ...because LLMs are really good at writing tests.
No, they're absolutely shit at writing tests. Writing tests is mostly about risk and threat analysis, which LLMs can't do.
(This is why LLMs write "tests" that check if inputs are equal to outputs or flip `==` to `!=`, etc.)
I think LLMs are really bad at writing tests. In the good old days you invested in your test code to be structured and understandable. Now we all just say "test this thing you just generated".
I shipped a really embarrassing off-by-one error recently because some polygon representations repeat their last vertex as a sentinel (WKT, KML do this). When I checked the "tests", there was a generated test that asserted that a square has 5 vertices.