LLM pretraining might be viewed as learning useful state representations from sequences, rather than having humans explicitly design the states in advance.

In many conventional software systems, we define states and transitions ourselves: idle, sending, completed, error, and so on. LLM pretraining works differently: the model learns from large numbers of sequences to predict what is likely to come next.

I don’t mean that an LLM literally constructs discrete internal states. Rather, it learns latent representations of context that are useful for prediction. In that sense, these representations might be thought of as a kind of learned state.

This seems closely related to abstraction. Abstraction is not simply throwing information away, but abstracting away differences that are irrelevant to prediction while preserving those that matter.

Two sequences may look very different on the surface, yet rely on similar underlying features to predict what comes next. Even a rare feature should be preserved if it changes the prediction.

From this perspective, pretraining can be seen as a form of useful compression: learning representations that preserve what matters for prediction while abstracting away what does not.

How far can abstraction, compression, and prediction be understood as different aspects of the same process?