Thanks for the comment - you're right, of course.

I read a bit more into the history of fp32 master copies, like in Mixed Precision Training (Micikevicius et al., ICLR 2018, https://arxiv.org/abs/1710.03740), it says that an fp16 update is lost once the weight is ~2048× larger than it. I've updated the post with a citation to that paper.

bf16's shorter mantissa moves that to ~512×, which is significant.

The point of the post is narrower: a lot of published fine-tuning code doesn't keep a master copy at all. So what does that do? It gives you a magnitude mask where only the smallest weights (those under 512xlr) are updating. Which I thought was an interesting inductive bias that wasn't really noted.