alt Hacker News> It programs with the skill of the median programmer on GitHub
This is a common intuition but it's provably false.
The fact that LLMs are trained on a corpus does not mean their output represents the median skill level of the corpus.
Eighteen months ago GPT-4 was outperforming 85% of human participants in coding contests. And people who participate in coding contests are already well above the median skill level on Github.
And capability has gone way up in the last 18 months.
Replies
Trying to figure out how to align this with my experiences (which match the parents’ comment), and I have an idea:
Coding contests are not like my job at all.
My job is taking fuzzy human things and making code that solves it. Frankly AI isn’t good at closing open issues on open source projects either.
> The fact that LLMs are trained on a corpus does not mean their output represents the median skill level of the corpus.
It does, by default. Try asking ChatGPT to implement quicksort in JavaScript, the result will be dogshit. Of course it can do better if you guide it, but that implies you recognize dogshit, or at least that you use some sort of prompting technique that will veer it off the beaten path.
The best argument I've yet heard against the effectiveness of AI tools for SW dev is the absence of an explosion of shovelware over the past 1-2 years.
https://mikelovesrobots.substack.com/p/wheres-the-shovelware...
Basically, if the tools are even half as good as some proponents claim, wouldn't you expect at least a significant increase in simple games on Steam or apps in app stores over that time frame? But we're not seeing that.