Definitely agree with this. Even without a large backlog, one of the things I find working on my personal project/product where I'm simultaneously the engineer/designer/project manager is it's really easy to ask the LLM to implement an idea I've been mulling for an hour or two, it one-shots it and I'm happy, and then a week or two later it starts to dawn on me that the feature was maybe not a great idea. Which isn't on the LLM, but I know when I write features by hand I tend to reach the "this is a bad idea" conclusion a lot earlier because I'm directly confronted with the cases where it won't work out, I have to think a lot harder about corner cases, etc.
Where I think/hope this goes is instead of using LLMs to go faster, we use them to do better work. I'd rather someone vibe code up better ways to test things, or use it to do in-depth code analysis and bug fixing, etc., then just pile in features.