alt Hacker News> bf16 can only represent changes to weights if the change is at least ~1/512 of the weight itself, so at small learning rates only the smallest weights can update.
Isn't the standard numeric fix for this to probabilistically update the weight? That is, if the desired change is current_value * 1.0014, it should have a 70% chance of going to the successor, and a 30% chance of staying unchanged. Over a number of steps, this will adjust it appropriately over time.
Yep, Stochastic Rounding [1]. I actually had "round-to-nearest" in the title of the piece originally. Usually people do that if they don't have the space for fp32 master weights, but it doesn't fix the second moment part of the learning process.
[1] What is New in Stochastic Rounding: a Survey on Theory, Hardware, and Applications, https://arxiv.org/pdf/2603.06060v2