alt Hacker NewsThough, I would point out that where people fall on that seems to correlate very highly with their ability to explain how an attention head works.
Replies
beardedwizard • today at 3:29 PM
Explain it to me
Though, I would point out that where people fall on that seems to correlate very highly with their ability to explain how an attention head works.
Explain it to me
Which direction is the correlation?
I don’t think this phrase means what people assume when it’s applied to post trained instruct models - which did not exist when the paper was written.
After RL it is not predicting based on samples of the original corpus - but is also chasing a reward function that does require other features.
There has been a lot of subsequent research that really calls many of the statements in this article into question.