I think a big issue is that the companies pushing AI want it to replace people entirely. So the software around it is designed with that mindset.

I'd really like something that works more like pair programming. Where you share an editor session with the AI, and it's much more interactive. eg. It says "I'm thinking about doing X here, what do you think?". Then you give a response and it continues. Or you can see it doing something silly and immediately stop it and correct something either with a prompt or by manually editing yourself, before letting it rip again. Or just ask it "why are you doing it that way?". Maybe with some control over the speed it's going, for when you feel confident about what it's doing.

Claude has made some positive changes over time where it asks you more when there are different approaches it can take. But I feel like I'm not being brought along with my mental model as much as I would like. A lot of the time it spits out a whole pile of code and then I have to go and build up the mental model after the fact, and correct a lot of what it's done.

The current approach is good a lot of the time though, when you're not really changing anything architectural and just want it to bash out code while you do somehthing else.