alt Hacker News0.84 Spearman fidelity to the MiniLM teacher at ternary precision is a striking result. How much of that is the quantization-aware training doing the work, versus what a post-training ternary quant of the same encoder would give you?
0.84 Spearman fidelity to the MiniLM teacher at ternary precision is a striking result. How much of that is the quantization-aware training doing the work, versus what a post-training ternary quant of the same encoder would give you?
It's entirely the QAT. The whole distillation process is quantization-aware from the start, so the ternary weights are learned rather than fitted after the fact.
The only post-training quantization I applied was int4 on the embedding layer, and I ran a small ablation there to find the sweet spot between size and quality.