alt Hacker NewsEvidence from listening to slopcoders speak says they feel it's night and day. Evidence from observing reality shows a 20% speed increase at best and 50% speed decrease at worst and almost always a significant reduction in quality.
Replies
smcg • yesterday at 1:49 PM
Twitter is a rather crude place, but it can be succinct. An exchange I saw recently:
A: "I spent more tokens than the entire OpenCode team combined."
B: "Why aren't you more successful?"
➕ show 1 reply
I was also skeptical, but I started measuring quality (I'm using rework as a rough metric) and quantity (PR throughput), and I've found that, on average, LLMs are 60% worse at writing code (meaning they rework at a rate 60% higher than humans) but my team produce PRs about 2.4x faster. So objectively my team's numbers don't match up with yours: our speed is about 2.5x increase _on average_ and a 60% reduction _on average_ in quality. Some basic math suggests that our ROI is something around 50-60% improvement over just humans.
I'm not convinced that throwing more money (i.e. tokens) at the problem would make us any more efficient, but 1.5x improvement ain't nothing.
And just to address the "night and day" bit.. if you gave a construction foreman a machine that made his team 20% more productive, he'd take that all day long and tell everyone it's "night and day." I'm not sure what your threshold is, but +20% isn't nothing either. That foreman would hit more deadlines than not, and get more bonuses than not.