>In contrast to a classic RNN, there's no unbounded hidden state accumulating across an entire trajectory

I don' understand this line. In a classic RNN hidden state is bounded dimension. In fact it's transformers that technically have unbounded hidden state.

You can't parallelize classic nonlinear RNNs for various reasons but in training both RNN and Transformer depend on the entire sequence history in a way that is unbounded. Of course in practice you just train on a max sequence length.

RNN xhat[t+1]=f(x[t],h[t])

Transformer/self-attention xhat[t+1]=f(x[t],h[t],h[t-1],...,h[1])