Unless something dramatic changes and it'll become cheap to "constantly re-educate" the models to learn the new fundamental facts about the system (ideally with a small lag of maybe a few days like with humans) I see no reason to think that LLMs can have a profound understanding of how business systems work of a remotely comparable quality to humans. Just adding the context about the system already reduces it's capacity significantly according to what I can tell. Not having large context helps somewhat at a cost of having to explain lots of task-relevant things to LLM over and over again...

> The agents only have to be better than we are and we’re not very good. Same argument as self-driving.

Self-driving requires less holistic thinking than imagining, building, and evolving large scale systems. It requires abilities that LLMs do not have. It requires non-symbolic reasoning, intuition, etc. that are beyond LLMs.

But an appropriate level of caution about change is necessary. Change isn't always good https://www.mentalfloss.com/science/environment/clair-patter... , and it doesn't always deliver what it promises or arrive on schedule.

> The agents only have to be better than we are and we’re not very good. Same argument as self-driving.

> Don’t fear change.

Change is inevitable.

What I understood from the comment is that Op fears the boiling frog scenario. [0]

0 https://en.wikipedia.org/wiki/Boiling_frog

Imagine a country where no one can drive, and all transportation relies on a few AI monopolies.

If you don’t pay them, you can walk by feet. If they ban you, you can walk by feet. If their infrastructure is damaged, no one and nothing will be able to move.