> Much of an LLM's capability comes from the structure encoded in its learned representations

And thats encoded as a set of next token predictions. So the way to see how reliably it solves a problem is to look at the chain of predictions, and see where it is unreliable at finding the next spot, or where it always fails and you need to add that link to the dataset to train it.

This isn't magic, today we understand pretty well how to add new skills to LLM, and the better this is understood the faster progress will be.

This also means that if a context doesn't have any good predictions, it will produce a dumb prediction for that context. This results in these bad outcomes, because currently LLM doesn't have a map for where predictions are good or bad.