I've been on a similar path, and I used to put time in high level design, as it was basically required in some cases. But not a lot of time, because smaller agents research, perform small tests, whatever to inform the high level design. Working incrementally seems to help, maybe prompting a bit more often, but the same can be accomplished with well defined checkpoints.

Then generalize + standardize that process, get an agent to replace you as high level designer so you can manage a team of high level designers, etc etc