alt Hacker NewsAs far as I know, the last layer is very quantization-sensitive, and is typically not quantized, or quantized lightly.
Have you experimented with having it less quantized, and evaluated the quality drop?
Regardless, very cool project.
As far as I know, the last layer is very quantization-sensitive, and is typically not quantized, or quantized lightly.
Have you experimented with having it less quantized, and evaluated the quality drop?
Regardless, very cool project.
(Not OP)
It depends on the model, but from my experiments (quantizing one layer of a model to 2-bit and then training the model with that layer in 2-bit to fix the damage) the first layer is the most sensitive, and yes, the last layer is also sensitive too. The middle layers take the best to quantization.
Different components of a layer also have a different sensitivity; e.g. the MLP downscale block damages the model the most when quantized, while quantizing the Q projection in self attention damages the model the least.