alt Hacker News> 1 bit per weight
does this basically correspond to moving each weight either up or down by a fixed amount? I'm a bit surprised you don't at least need a "stay same" bit, but i suppose it could balance out over multiple iterations.
Interesting that it works at all. Although, thinking on it, I could see it maybe even having a nice regularizing effect where every layer would end up have similar weight magnitudes. (like projecting onto the local n-ball as mentioned in a paper posted recently on HN)
Replies
f_devd • 10/12/2024
It has been more formally studied in signSGD[0], and empirically it's comparable to Adam in terms of behavior.
This is for keeping the weight vectors in sync between two machines.
The weight vectors themselves are regular floats. But the data exchanged between the machines is 1 bit. Basically, you keep track of changes to the weight vector which hasn't yet been propagated to the other machine. You quantize this to 1 bit per weight (ie. a sign bit) and send it, together with a single scale factor X, accumulating the quantization error for the next sync iteration.
You choose X to be the RMS or some similar metric of the accumulated error.