OP, you accidentally invented the design process that they teach in design school. You can officially call yourself a designer.
1) have a high level functional goal
2) be inspired and think of a motif
3) go wide within that motif and explore
4) narrow in on a single direction and iterate
5) fine tune the details until you are creatively satisfied and the design meets your goals from step 1.
Yeah, and the trade-off here is that it takes time. In the same way, you can write insightful blog posts or work emails using AI if you maintain creative control, but all of sudden, you're not moving at 100x speed. You're moving a lot closer to human speed.
This trade-off is something I like the least about AI. In the time it takes someone to produce excellent results using AI without giving up control, people unconcerned with quality or style can produce 100x as much. So by the end of the day, your inbox or your social media feed is mostly slop.