logoalt Hacker News

xnorswap • today at 12:49 PM • 0 replies • view on HN

It's not the per-run costs I worry about, it's the costs of churn.

As soon as Oracle Tests fail, the LLM has three options:

1. Rewrite the test - This undoes the benefit of the test in the first place 2. Implement workarounds and quirks to match the oracle, often at the expense of recreating buggy behaviour 3. Implement both corrected behaviour and quirks-mode behaviour to meet the oracle, raise further jobs to formalise the switch over and change of behaviour.

( And this "change of behaviour" might be a few pixels out of place! Or it could be a much more significant change. When trying to meet a goal, LLMs will often treat these with the same zeal. )

If you're working interactively, you can catch it and steer it to what's best for the feature. If you're fully automating development, then you're at the mercy of whatever the model does at that moment.

Oracle-based testing has its place, but I don't think it's good for large scale automated testing.

Also, you're talking about $500/release 2-3 times a week. That's fine, that's less than the cost of an employee.

Anthropic and others are talking about how to scale to merge 2000-3000 PRs per week.

At that scale, that cost per merge is $6M/month, surely a cost you'd want to minimise if you can cut them by half with better type-checking, static analysis, and other cheap deterministic tests, even if that's still a small cost of the whole project.

If nothing else, it's pure waste. Our planet is (mostly metaphorically) burning, and I believe that more efficient IT processes and development practices would go a long way to helping mitigate some of that.