alt Hacker NewsIt's all about the test suite. The test suite becomes an executable specification, and the better the spec, the better the results you can get from AI.
Replies
The test suite is the same thing as the code, just approached from the other side.
i want the code to do this --> write the code to do this
i want the code to do this --> write the tests to specify this
the hard part is not the code or the tests. it's knowing what 'this' is supposed to be, exactly.
if you have the code, it is trivial to write the tests that will perfectly test the code. if you have the tests, it is trivial to write the code that will perfectly pass the tests.
however, none of this has anything to do with whether the code or the tests are good, and having a bad version of one will not help you write a good version of the other.
It really isn't.
It's s about subpar models trained on subpar data doing subpar work.
The only reason why anyone takes it seriously is that we've had a glut of subpar developers for 30 years so it all balances out in the end.
How do you write a spec for correctness? Only the small and unimpressive programs can be checked exhaustively.
It's part of it but you need good systems thinking in order to write good tests. The kind of thinking which requires years of software engineering experience to acquire.
Also the architecture really matters now because a poor architecture will lead to more weird edge cases and require more exotic tests to plug all the holes.
It's all now about money extraction from the areas dependant on software, not the sofrware itself.
It doesn't sound very intelligent to just brute force iterate until you pass all the tests. This sounds like what Rich Hickey called "bashing into the guard rails". Even if we assume the tests suite is perfect and will guide the model to a correct solution, maybe this is one case we really should just say "LLM" rather than "AI".
If you have such a test suite then you really don't need AI to write the code for you.
The very first thing many people did with AI is start calling the automated tests it writes good enough to capture the desired behavior. Writing tests isn't a whole lot of fun for most, so now the thing that it is supposed to help ensure we're not evolving our software into piles of trash is, in fact, one of the most neglected parts of vibed out codebases.