logoalt Hacker News

kpw94 • today at 4:33 PM • 1 reply • view on HN

> About the generation speed: ~100-150 t/s on the RTX 5090 and ~40 t/s on the Mac

Curious if you can share the prefill speed too?

I run locally on a crappy desktop (some AMD iGPU with Vulkan llama.cpp, 32 GB DDR4 RAM) for experimentation. I get 15 tok/s on generation for the qwen & gemma4 MoE models. I get around 150 tok/s prefill speed.

Reason I'm asking about the prefill is looking at my stats at work, I use between 20M to peaks of 300M input tokens daily. Some of those token are cached but in general, I seem to have roughly 500x more input tokens than output. So interested in prefill tok/s stats.

Huge Thank you for llama.cpp btw!!

Replies

ggerganov • today at 4:58 PM

Here are the prefill speeds:

    Device 0: NVIDIA GeForce RTX 5090, compute capability 12.0, VMM: yes, VRAM: 32109 MiB
  | model                          |       size |     params | backend  |  fa |            test |                  t/s |
  | ------------------------------ | ---------: | ---------: | -------- | --: | --------------: | -------------------: |
  | qwen35 27B Q4_K - Medium       |  15.92 GiB |    27.32 B | CUDA     |   1 |   pp2048 @ d512 |      3714.02 ± 10.85 |
  | qwen35 27B Q4_K - Medium       |  15.92 GiB |    27.32 B | CUDA     |   1 |  pp2048 @ d1024 |      3684.86 ± 15.21 |
  | qwen35 27B Q4_K - Medium       |  15.92 GiB |    27.32 B | CUDA     |   1 |  pp2048 @ d2048 |       3650.80 ± 8.53 |
  | qwen35 27B Q4_K - Medium       |  15.92 GiB |    27.32 B | CUDA     |   1 |  pp2048 @ d8192 |       3473.88 ± 0.97 |
  | qwen35 27B Q4_K - Medium       |  15.92 GiB |    27.32 B | CUDA     |   1 | pp2048 @ d32768 |       2754.69 ± 4.07 |

  ggml_metal_device_init: GPU name:   MTL0 (Apple M2 Ultra)
  | model                          |       size |     params | backend  | fa |            test |                  t/s |
  | ------------------------------ | ---------: | ---------: | -------- | -: | --------------: | -------------------: |
  | qwen35 27B Q8_0                |  26.62 GiB |    26.90 B | MTL      |  1 |   pp2048 @ d512 |        379.75 ± 0.21 |
  | qwen35 27B Q8_0                |  26.62 GiB |    26.90 B | MTL      |  1 |  pp2048 @ d1024 |        377.15 ± 0.35 |
  | qwen35 27B Q8_0                |  26.62 GiB |    26.90 B | MTL      |  1 |  pp2048 @ d2048 |        371.46 ± 0.91 |
  | qwen35 27B Q8_0                |  26.62 GiB |    26.90 B | MTL      |  1 |  pp2048 @ d8192 |        344.84 ± 0.41 |
  | qwen35 27B Q8_0                |  26.62 GiB |    26.90 B | MTL      |  1 | pp2048 @ d32768 |        222.42 ± 5.29 |
Btw, based on your numbers, I think our use cases are quite different. I use the agent for very targeted sessions - basically things that are clear to me how to do, just want to automate them. My workflow is usually: new session -> read this, this and this -> do that. I.e. I don't let it wander at all in the codebase, so I rarely exceed the context window.

Also, I get a lot of mileage from the ngram-based speculative decoding functionality [0] as it allows me to iterate on the implementation much faster.

[0] https://github.com/ggml-org/llama.cpp/pull/19164

➕ show 1 reply