The AI Productivity Gap

The outwardly measurable productivity increase will come but over a time scale of years, and most of that will be in barely measurable increments. Nobody wants to hear that but it's just how the world works.

Writing code is a small part of everyday's job of a software engineer. The article's table reflects this fairly well.

AI compresses implementation time for an individual engineer, but architecture decisions, design reviews, integration, testing, deployment, and production validation remain largely serial activities. If code generation speeds up by 5x while those bottlenecks don't, you've mostly increased the team's work queue rather than its throughput.

With the current capabilities, models still need constant babysitting and course correction. An engineer who lacks the skills to guide them can end up creating more work for the rest of the team. AI makes it easy to generate code faster than you can understand it, and that cost is paid during code review, debugging, and maintenance by colleagues, whose confidence in that engineer's skills may be affected by his use of AI.

What looks like a productivity gain for one engineer can become a productivity loss for the team as a whole.

Pre AI and Post AI code review hours are both 0.75 in this made up example. I find that implausible.

Even with the same amount of code, AI code is less trustworthy* and requires more attention... but we know it won't be the same amount, it will be more. This means it will take longer to review, or there will be unforeseen consequences of not spending that extra time.

*meaning no human eyes have looked at it and said "this doesn't make sense", or "this is cheating", or "this doesn't meet requirements", and won't be caught until code review if at all.

How to get rid of every highly-skilled-but-unmedicated neuroatypical developer (could be people like Xe Iaso or Soatok):

> hiring someone who is a good coder, but has trouble reasoning about systems, has no patience for working through hard problems with others, and can’t break down vague requirements into tangible action items.

Why not hire the excellent developers for the highly-technical skills they bring, and match them with architects/product managers who are the ones who have the big picture? Am I crazy to think like this?

One extra consideration that is material.

Familiarity with what's already existing affects how much AI helps as a research tool.

A new developer to the team can research quickly using AI, while by contrast, experienced folks won't gain much vs. just using their current knowledge.

It shows up as a new team member coming up to speed impressively fast. But since it doesn't help the rest of the team it also doesn't contribute much to overall team productivity.

What I have noticed in my own work that a lot of the time that used to be for coding is now just waiting. I have three agents working on three different features in parallel, and I'll go back and forth with all of them, correcting things and steering etc, but then I find myself with three busy agents and nothing to myself except stare at the screen while they code away. There is a mental budget for me where I can't have more than those three running at the same time and still keep track so what I end up doing is just scrolling HN...

The compensation gap, to hell with more 'productivity'; for who? Not myself or even the business, the last time I worked this hard to achieve (earn) so little, I was a teenager.

It doesn't really matter how much more productive a developer is if all other roles at the company don't follow suit. Before a developer picks up something to work on a series of roles had to set their eyes on work to be done. Project/product leads, tech leads, business people stamping and deciding on priorities. Then there's all the work that happens after a developer finishes work which tends to be manual as well. Review, QA, education, ops changes, marketing material, education articles, webcasts, showcasing features to end users and lets not forget end users actually making good use of the amazing new features shipped and likely many more largely sequential processes depending on company size and product/project type.

There's no real way to get to a 10x developer nowadays. Even if a company somehow achieved the magic productivity increase in all employees you still need a 10x consumer to gulp it all down.

> There’s no doubt that AI has already improved the productivity of engineering teams, and will only get better in the coming years.

They lost me by begging the question in the very first sentence.

Based on personal observation, a lot of productivity has been thrown out of the window with unneeded refactoring, rewrites and "what-if" scenarios that the AI agent will spot.

I've had similar conversations with a client recently while discussing estimates for a large project. Senior leadership has a mental model where AI makes everything X% faster, but that's very wrong. Some things get sped up by an insane amount and basically go to zero, some others not so much. Entirely new tasks emerge, such as directing agents to provide them the context they need, setting loops, etc.

It's a very O-ring problem.

> There’s no doubt that AI has already improved the productivity of engineering teams

Thought it might be an interesting read, however have up just after reading the first line.

For the context, code had always been a copy-paste exercise, big part of it was understanding and differentiating between the different choices. Along with it people were growing as engineering practitioner's too. Human learning still needs to happen if they are expected to fix the code when LLM gives up.

LLMs are quite useful tool in themselves, however the hype has unfortunately polarized the population.

AI is a force multiplier.

It multiplies both good and bad decisions. Both mine and it's 'own'.

I can get some things done 10x faster and it might even catch mistakes or help me solve something difficult.

But if I am being lazy or complacent then it bites me that much harder.

The role of developer is becoming more of a system thinker than a syntax writer, previously load balance was about what to do in a given timeline, now its more of what not to do , doing more is actually increasing technical debt.

I don’t think human review is worth it for LLM generated code. We design abstractions and all around how humans think. LLMs writes code that is better understood by machines. If you are all in on LLMs, by all means, read the code figure out what it means. But trying to enforce a human flow to its logic is flawed and will be overwritten the next time.

Based on personal experience on a specific project, that 1.5 hours with AI let me accomplish work planned for a man-week in the pre-AI era. So it’s much more than 3x.

"We can disagree about the specific numbers here, but if you think this is wildly off, you’ve probably never been a senior developer"

I'd even say the productivity gap is even smaller, if not negative in some areas...

It's like you just made up those numbers and then developed your thesis around that.

If a senior developper spends the same amount of time debugging, code reviewing, setting up CI/CD, documenting and doing admin work, he has not been using AI right.

This assumes you are arranging deckchairs and not leaving the cruise ship for say, a boeing 747.

One example, let's say there is a side bet that makes everyone 10x more productive with a success rate of 1%

It takes 2 hrs to make the bet wit agent orchestration.

10 people can get this done in their spare time freed up by AI in 5 weeks.

Bet cashes in and you are much faster at everything.

It won't feel faster. Because the brain probably scores emotionally in roadblocks cleared per hour.

Back when you got a single punch card loaded in a day it felt like a fucking win.

The other factor is you get paid the same and there is more disruption and competition and job insecurity.

But objectively value gets shipped faster using AI.

Just not much if you go the faster horses route with AI. You need the cars. (Or planes!)

> Reading and Debugging 1.5 1.0 > Code Reviews 0.75 0.75

Since these numbers are made up, I may as well throw my personal anecdote in the ring. I find reading and reviewing far harder with coworkers who are using AI. Tickets contain about 5x as much meaningless junk as they used to, and testing notes - while far more thorough - are often now multiple pages in length. Reviews also contain much more code, people try to do more drive-by fixes because the models can generate those fixes so quickly, and people understand the code they're submitting far less clearly because the model is able to generate fixes they simply couldn't previously.

I feel less productive than I was a year ago, and I don't see my team shipping more features than they were previously. But everyone reports that they're far more productive. I don't get it.

> Sometimes I actually find AI makes non-coding work go slower…For now, let’s assume AI only helps.

Haha fuck you dude no I’m not going to assume it only helps when your preceding paragraph gives a concrete example of HOW IT MAKES WORK LESS EFFICIENT.

In turn, I’m not “assuming” this guy is delusional and “grasping at straws” I’m deducing it from his poorly constructed, self-defeating, fictional argument in favor of his assertion.

[dead]

[dead]

[flagged]

[flagged]

[dead]