Ok, so take the example of RF communications:

You need to send data, but only 50% at random comes through. You could move things closer together, send at higher power, improve the signal modulation to give you higher snr.

All of these things are done, this is improving the base system. But at the end of the day rf is in the real world, and the real world sucks. Random shit happens to make your signal not come through.

So what do you do? You design fault tolerant systems.

You add error correction to detect and automatically fix errors on receive.

You add tcp, to automatically retransmit missed packets.

You add validation to make sure that the received data is sane.

You use ecc ram to validate that an ion from the sun has not bit flipped your data.

These are extremely complex hierarchies of systems, and fundamentally they are unreliable. So you design the whole thing with error margins and fall back handling.

Yes a better understanding of the underlying system in question does allow you to make more efficient error correction/validation, but it does not change the fact that you need error correction.

And in the case of RF signals, for example, the most optimal design is not zero errors. In fact they way you design it is so that for a given snr, you expect a given probability of error. To get the maximum throughput, you design your error correction coding to handle that. And because even with that, there can be edge cases, you design higher level Mac layer logic to resend.

Yes, a better understanding of the error cases will make the error correction (agent loops) in llms better over time, but it will not remove the error correction.

there may be situations similar to RF, where a less accurate, but faster model, with more validation is preferred for a variety of engineering reasons, either throughput/cost/creativity/etc.

You need to look at llms as black boxes of the physical world, that we question, and get a answer. At the level of complexity it is more akin to physics, than software, there is no reliability inherent to them, only what we design around.

And there are tons of people doing the fundamental physical research into how they work, and how to make them work better.

But that is a completely different avenue of research from how to make useful systems out of unreliable components.

You do not need to fully model the path propagation of em waves in order to make a reliable communication system, it might help, but it will be fragile if you solely rely on it.

And if you engineer a reliable system architecture, it does not get fully invalidated when the scientists build better models of the underlying systems. You may modify the error correction codes but they do not go away.

For example the original Morse code, had hand handmade validation checks, handshakes etc. the architecture did not get wholly replaced with the advent of Shannon's information theory, even if some of the specific methods did.

And WW1/WW2 telegraphs, and communications were still useful despite not understanding the underlying information theory and being unreliable in many situations.