alt Hacker NewsI get ~5 tokens/s on an M4 with 32G of RAM, using:
llama-server \
-hf unsloth/Qwen3.6-27B-GGUF:Q4_K_M \
--no-mmproj \
--fit on \
-np 1 \
-c 65536 \
--cache-ram 4096 -ctxcp 2 \
--jinja \
--temp 0.6 \
--top-p 0.95 \
--top-k 20 \
--min-p 0.0 \
--presence-penalty 0.0 \
--repeat-penalty 1.0 \
--reasoning on \
--chat-template-kwargs '{"preserve_thinking": true}'
I haven't tested the 27B model yet, but 35B-A3B often gets off rails after 15k-20k tokens of context. You can have it to do basic things reliably, but certainly not at the level of "frontier" models.
Replies
Why use --fit on on an M4? My understanding was that given the unified memory, you should push all layers to the GPU with --n-gpu-layers all. Setting --flash-attn on and --no-mmap may also get you better results.
When you say tok/s here are you describing the prefill (prompt eval) token/s or the output generation tok/s?
(Btw I believe the "--jinja" flag is by default true since sometime late 2025, so not needed anymore)
How is the quality of model answers to your queries? Are they stable over time?
I am wondering how to measure that anyway.
Using opencode and Qwen-Coder-Next I get it reliably up to about 85k before it takes too long to respond.
I tried the other qwen models and the reasoning stuff seems to do more harm than good.
We also made some dynamic MLX ones if they help - it might be faster for Macs, but llama-server definitely is improving at a fast pace.
https://huggingface.co/unsloth/Qwen3.6-27B-UD-MLX-4bit