Same here, and I also really enjoy the high level design/structure part of it.
THAT part doesn't mesh too well with AI, since it's still really bad at autonomous wholistic level planning. I'm still learning how to prompt in a way that results in a structure that is close to what I want/reasonable. I suspect going a more visual block diagram route, to generate some intermediate .md or whatever, might have promise, especially for defining clear bounds/separation of concerns.
Related, AI seems to be the wrong tool for refactoring code (I recently spent $50 trying to move four files). So, if whatever structure isn't reasonable, I'm left with manually moving things around, which is definitely un-fun.
Definitely go for that middle step. If it's something bigger I get them to draw out a multi-phase plan, then I go through and refine that .md and have them work from that.