There are definitely issues in human software engineering which reach some combination of the following end states:

1. The team is unable to figure it out

2. The team is able to figure it out but a responsible third-party dependency is unable to fix it

3. The team throws in the towel and works around the issue

At the end of the day it always comes down to money: how much more money do we throw at trying to diagnose or fix this versus working around or living with it? And is that determination not exactly the role of a product manager?

I don't see why this would ipso facto be different with AI

For clarity I come at this with a superposition of skepticism at AI's ultimate capabilities along with recognition of the sometimes frightening depth encountered in those capabilities and speed with which they are advancing

I suppose the net result would be a skepticism of any confident predictions of where this all ends up