alt Hacker NewsThe entire issue arises from the use of truncation, right? It guarantees that only an exact 1.0 could land in the 255 bin so the net effect is a reduction of 256 bins to 255 bins. (Using random numbers as shown also guarantees no 1.0.)
Why not scale to fill the available bins, though? i.e. trunc(result * 255.999)?
> Why not scale to fill the available bins, though? i.e. trunc(result * 255.999)?
That’s half of the mid-riser staircase quantizer discussed in the article. (The other half is coming up with the reverse.)
(I would implement it as min(floor(x * 256), 255).)