alt Hacker NewsThat's what I meant - we are currently use fp4 formats for training, and we cannot quite get away with that, despite dynamic quant and small block size - we still have to use quite a bit of higher precision (fp8 or even fp16) in various model components.
I might still be misunderstanding what you are saying, but bitnet also keeps high precision latent weights during training. The optimizer updates those, while the weights used in the forward pass are quantized to ternary values.