> will there be a day that AI will know what are "the right things to build" and have the "agency" (or illusion of) to do it better than an AI+human
I share this sentiment. It's really cool that these systems can do 80% of the work. But given what this 80% entails, I don't see a moat around that remaining 20%.
I can't explain it in a way that will make sense or base it on any data, but I do see that moat all the time. For example, Microsoft / GitHub somehow taking a 3 year lead (with Co-Pilot) and losing it to Cursor in months, and still catching up.
Microsoft / GitHub have no real limitation to doing better/faster, maybe it's the big company mentality, moving slower, fear of taking risks where you have a lot to lose, or when the personal incentive for a product manager at github is much much lower than the one of a co-founder of a seed stage startup. Co-Pilot was a microscopic line item for Microsoft as a whole, and probably marginal for GitHub too. But for Cursor, this was everything.
This is why we have innovation, if mega-corps didn't promote people to their level of incompetence, if bureaucracy and politics didn't ruin every good thing, if private equity didn't bleed every beloved product to the last penny, we would have no chance for any innovation or entrepreneurship because these company have practically close to unlimited resources.
So my only conclusion from this is - the moat is sometimes just the will to do better, to dare to say, I don't care if someone has a billion dollars to compete with me, I'll still do better.
In other words, don't underestimate the power of big companies to make colossal mistakes and build crappy products. My only worry is, that AI would not make the same mistakes an we'll basically have a handful of companies in the world (the makers of models, owner of tokens e.g. OpenAI, Anthropic, Google, Amazon, Meta, xAi), if AI led product teams will be able to not make the mistakes of modern corporations of ruining everything good that they got in their hands, then maybe software related entrepreneurship will be dead.