alt Hacker NewsThere's a reason we keep around an fp32 copy of the weights?
This issue was recognized a decade ago when we first started training with fp16.
There's a reason we keep around an fp32 copy of the weights?
This issue was recognized a decade ago when we first started training with fp16.
Thanks for the comment - you're right, of course.
I read a bit more into the history of fp32 master copies, like in Mixed Precision Training (Micikevicius et al., ICLR 2018, https://arxiv.org/abs/1710.03740), it says that an fp16 update is lost once the weight is ~2048× larger than it. I've updated the post with a citation to that paper.
bf16's shorter mantissa moves that to ~512×, which is significant.
The point of the post is narrower: a lot of published fine-tuning code doesn't keep a master copy at all. So what does that do? It gives you a magnitude mask where only the smallest weights (those under 512xlr) are updating. Which I thought was an interesting inductive bias that wasn't really noted.