logoalt Hacker News

trouve_search • yesterday at 7:18 PM • 3 replies • view on HN

What configuration are you using? On both vllm and llama-cpp, I get significantly higher speeds from gemma4 than qwen3.6 (with their respective speculative decoding methods).

Output TPS in vllm for instance:

- Gemma4 26B-A4B: 200-300TPS

- Qwen3.6 35B-A3B: 120-180TPS

- Gemma4 31B: 80-120TPS

- Qwen3.6 27B: 60-80TPS

This is for a first request on a dual 5090 setup, with their respective speculative decoding methods.

Replies

mirekrusin • today at 12:15 AM

Dual 4090, getting 85-113 t/s depending on task (draft seems to speed up quite a lot, disproportionately more for content like svg etc):

  ./llama.cpp/llama-server \
        -hf unsloth/Qwen3.8-27B-GGUF:UD-Q8_K_XL \
        --webui-mcp-proxy \
        --no-mmproj \
        --parallel 1 \
        --kv-unified \
        --flash-attn on \
        --fit off \
        --split-mode tensor \
        -ngl 999 \
        --cache-type-k q8_0 \
        --cache-type-v q8_0 \
        -ub 256 \
        --no-context-shift \
        --host 0.0.0.0 \
        --tools all \
        --jinja \
        --ctx-size 262144 \
        --spec-type draft-mtp \
        --spec-draft-n-max 3 \
        --reasoning on \
        --chat-template-kwargs '{"reasoning_effort":"medium"}' \
        --reasoning-preserve \
        --temp 1.0 \
        --top-p 0.95 \
        --top-k 20 \
        --min-p 0.0 \
        --presence-penalty 0.0 \
        --repeat-penalty 1.0
Use claude/codex/whatever with /goal to optimize params for you.

IMHO draft model support on dense models is great alternative to MoE on GPUs (high bandwidth, less memory) – more intelligence, speed somewhere mid way there which is usually sufficient.

petu • yesterday at 8:13 PM

Single 3090 under llama.cpp:

  | model               |    size |   test |  t/s |
  | ------------------- | ------- | ------ | ---- |
  | gemma4 31B Q4_0     | 16.1 GB | pp2048 | 1248 |
  | gemma4 31B Q4_0     | 16.1 GB |  tg512 |   40 |
  | qwen35 27B Q4_K     | 15.9 GB | pp2048 | 1248 |
  | qwen35 27B Q4_K     | 15.9 GB |  tg512 |   39 |
  | gemma4 26B.A4B Q4_0 | 13.3 GB | pp2048 | 4304 |
  | gemma4 26B.A4B Q4_0 | 13.3 GB |  tg512 |  160 |
  | qwen35 35B.A3B Q3_K | 15.7 GB | pp2048 | 3329 |
  | qwen35 35B.A3B Q3_K | 15.7 GB |  tg512 |  144 |
> with their respective speculative decoding methods

You're benchmarking drafter acceptance rate, then. Which is real life values, yes, but attributing worse drafter performance to the other 95% of the model being inherently slower.

xfalcox • yesterday at 10:28 PM

Have you tried running it on a single 5090? Dual 5090 require https://github.com/aikitoria/open-gpu-kernel-modules for higher perf. Are you using TP?