BERT isn't a next token predictor. It predicts a single token based on the whole surrounding context in both directions.
I mean sure, but BERT can be and often is used as a next token predictor / generator.
I could have used any series of NLP examples, the point is this CoT behavior only emerges when you get to a certain scale (and training style, presumably).
alt Hacker News
I mean sure, but BERT can be and often is used as a next token predictor / generator.
I could have used any series of NLP examples, the point is this CoT behavior only emerges when you get to a certain scale (and training style, presumably).