logoalt Hacker News

pama • today at 3:37 PM • 2 replies • view on HN

Thanks for this table! Do you happen to have the fraction of accepted speculative decoding outputs as well? I know it is asking much, but do you possibly measure differences in different across speculation techniques (dspark vs dflash) and differences in framework/code used (unless you only use in house code for these and cannot run sglang or vllm)? Your table is more interesting than the blog for sure.

Replies

scottcha • today at 4:05 PM

We do use spec decode on all these models (I don't know if the paper is using that but I'm assuming not given the models listed) but not all these models have integrated spec decode stats in the responses. Turns out GLM 5.3 does though so here is a pass on that model.

   Language      Accept   Tokens/step   0/3 steps   3/3 steps   Output tok/req
   English       76.53%      3.296         9.4%        62.8%            549
   Chinese       73.37%      3.201        11.3%        58.2%            754
   Russian       73.88%      3.217        11.1%        58.5%            739
   French        75.50%      3.265         9.8%        60.9%            631
   S. Pashto     70.44%      3.113        12.9%        54.5%           1332
   Tigrinya      79.72%      3.391         8.2%        67.8%           3643
   Shan          75.64%      3.269        11.2%        63.0%           3675
   Tibetan       74.67%      3.240        10.3%        60.2%           2857
   Overall       75.75%      3.273        10.3%        62.2%           1772
➕ show 1 reply
vova_hn2 • today at 5:18 PM

I think that the fact that TFA doesn't mention speculative decoding at all is not a good sign. It casts doubt both on the accuracy of the data and on the competence of whoever collected it.

I can imagine that in some cases the acceptance rate for non-English responses may be much higher, because while words on average need more tokens to spell, it is more "obvious" what the next token should be, so it is easier for the speculation model to predict it accurately. Information density per token is lower.

Or, maybe the acceptance rate is much lower, if the speculation model is tuned for English and doesn't work well with other languages.