He talks about the problem with this empirically.

I have to say I mainly agree in a way I wouldn’t have 1 month ago. We are now starting to see bugs creep above our baseline. This is only what we are detecting. At least in certain critical areas we are not removing strict human review.

One theory is our codebase (which is large and mature) was very coherent. As we moved to more autonomy and needed generated code in the base has grown and is less mature now and likely not as coherent. The LLM has to cope with this.