alt Hacker NewsAlong these lines: agents will write mountains of super repetitive test code, seemingly uninterested in refactoring it (without explicit instruction) to reduce redundancies.
In any non-trivial integration test scenario, it's easy to end up several times the LoC in tests compared to actual shipped code.
Second problem: different models, agents, and harnesses seem to have different tendencies and they tend to avoid touching code that is not their own unless explicitly instructed. What I mean by this: it will happily duplicate some helper method from another test file rather than pulling it out and reusing it, seemingly avoiding coupling (good kind in this case since model level changes should be propagated atomically).
Tests are where I see agents just spew absolute mountains of code.
Replies
Every senior I've ever worked with at some point embraced the idea of minimalism. Produce minimal changes with a minimal surface. Worked well at startups and especially well on large spaghetti legacy projects. Seeing all of a sudden now how suddenly number of kLocs, PRs, token maxxing, number of tasks in progress is again a metric with which non-engineers brag about on engineering conferences feels like the industry has reverted to some braindead pre-infant state. I guess all mistakes need to be relearned at every hype cycle.
When people have gotten excited about having so many tests, I've asked them what their plan is when 500 of their 1000 test suddenly stop passing. I've always favored 50 tests I know are meaningful, where any failure is something to respond to, over 500 tests with a bunch of overlap and edge cases I'm not sure matter. But now people aren't even thinking about that, they're just letting the agents make all those decisions.
I think this approach will continue to work for some people and orgs, but others are going to hit walls where their agents can't manage the mountains anymore. It's going to be interesting to see how people and orgs respond when that happens.