logoalt Hacker News

Unlocking Stable On-Policy RL on TPU with Bitwise-Zero Train–Inference Mismatch

5 points • by gmays • last Wednesday at 3:28 AM • 1 comment • view on HN

Comments

amluto • today at 4:03 AM

I think it would be nifty if someone convincingly solved this the other way: find a way for a model to learn well from the success/partial success/failure of a slightly off-policy rollout.

As a human, I certainly learn better by trying something myself (on-policy) than by watching someone else do it (very off-policy). But people, especially athletes, really do learn by watching videos of themselves trying something in the recent past (slightly off-policy), and I expect the main reason that watching the video doesn’t work even better than it does is that it’s just audiovisual and doesn’t come with all the sensations associated with the attempt. ML training doesn’t have that particular limitation.