I think the big question here is how effective can you make training and monitoring across a widespread population in practice?

In aviation, commercial pilots have very strict and extensive training and monitoring and as a result are generally able to utilize automation effectively while keeping up their manual skills. There are very rarely CFIT incidents in major commercial airlines.

The opposite is true in general aviation (small private Cessnas, etc), where it’s extremely common for pilots to buy more plane than they can handle and then rely on automation to bridge their skill gap. CFIT is much more common in general aviation, along with incorrect actions in response to real system failures that should have been recoverable. Automation complacency regularly kills in general aviation.

A key thing to notice is that automation isn’t outright prohibited in either commercial or general aviation, but there are distinct regulatory frameworks based on potential impact.

We accept looser rules for general aviation because the failures are societally less severe and because the population is much larger so effective training and enforcement would be significantly harder. In commercial airliners where failures are catastrophic, we have much stricter policies and require training and testing regularly to avoid automation complacency.

Will we start to see this practice in software? Probably, but only if/when the societal cost of NOT doing it becomes more clear. We regulated aviation because crashing planes are obviously bad. We license structural engineers because collapsing bridges are obviously bad. Will automation-induced software failures hit a similar tipping point?