The 27B model they release directly would require significant hardware to run natively at 16-bit: A Mac or Strix Halo 128GB system, multiple high memory consumer GPUs, or an RTX 6000 workstation card.

This is why they don’t advertise which consumer hardware it can run on: Their direct release that delivers these results cannot fit on your average consumer system.

Most consumers don’t run the model they release directly. They run a quantized model that uses a lower number of bits per weight.

The quantizations come with tradeoffs. You will not get the exact results they advertise using a quantized version, but you can fit it on smaller hardware.

The previous 27B Qwen3.5 model had reasonable performance down to Q5 or Q4 depending on your threshold for quality loss. This was usable on a unified memory system (Mac, Strix Halo) with 32GB of extra RAM, so generally a 64GB Mac. They could also be run on an nVidia 5090 with 32GB RAM or a pair of 16GB or 24GB GPUs, which would not run as fast due to the split.

Watch out for some of the claims about running these models on iPhones or smaller systems. You can use a lot of tricks and heavy quantization to run it on very small systems but the quality of output will not be usable. There is a trend of posting “I ran this model and this small hardware” repos for social media bragging rights but the output isn’t actually good.

I get ~5 tokens/s on an M4 with 32G of RAM, using:

  llama-server \
   -hf unsloth/Qwen3.6-27B-GGUF:Q4_K_M \
   --no-mmproj \
   --fit on \
   -np 1 \
   -c 65536 \
   --cache-ram 4096 -ctxcp 2 \
   --jinja \
   --temp 0.6 \
   --top-p 0.95 \
   --top-k 20 \
   --min-p 0.0 \
   --presence-penalty 0.0 \
   --repeat-penalty 1.0 \
   --reasoning on \
   --chat-template-kwargs '{"preserve_thinking": true}'

I haven't tested the 27B model yet, but 35B-A3B often gets off rails after 15k-20k tokens of context. You can have it to do basic things reliably, but certainly not at the level of "frontier" models.

I'm getting 30 t/s on RTX 4090D (using 42 out of 48GB VRAM) with UD-Q6_K_XL

https://huggingface.co/unsloth/Qwen3.6-27B-GGUF/discussions/...

There are infinite combinations of CPU/GPU capable of running LLMs locally. What most people do is buy the system they can afford and roughly meets their goals and then ball-park VRAM usage by looking at the model size and quantization.

For more a detailed analysis, there are several online VRAM calculators. Here's one: https://smcleod.net/vram-estimator/

If you have a huggingface account, you can set your system configuration and then you get little icons next to each quant in the sidebar. (Green: will likely fit, Yellow: Tight fit, Red: will not fit)

Further, t/s depends greatly on a lot of different factors, the best you might get is a guess based on context size.

One thing about running local LLMs right now, is that there are tradeoffs literally everywhere and you have to choose what to optimize for down to the individual task.

Qwen3.5-27B with a 4bit quant can be run on a 24G card with no problem. With 2 Nvidia L4 cards and some additional vllm flags, i am serving 10 developers at 20-25tok/sek, off-peak is around 40tok/sek. Developers are ok with that performance, but ofc they requested more GPU's for added throughput.

For Qwen3.5-27b I'm getting in the 20 to 25 tok/sec range on a 128GB Strix Halo box (Framework Desktop). That's with the 8-bit quant. It's definitely usable, but sometimes you're waiting a bit, though I'm not finding it problematic for the most part. I can run the Qwen3-coder-next (80b MoE) at 36tok/sec - hoping they release a Qwen3.6-coder soon.

As this is a dense model and it's pretty sizable, 4-bit quantization can be nearly lossless. With that, you can run this on a 3090/4090/5090. You can probably even go FP8 with 5090 (though there will be tradeoffs). Probably ~70 tok/s on a 5090 and roughly half that on a 4090/3090. With speculative decoding, you can get even faster (2-3x I'd say). Pretty amazing what you can get locally.

For Apple Mac, there is https://omlx.ai/benchmarks

These might help if the provider doesn't offer the same details themselves. Of course, we have to wait for the newly released models to get added to these sites.

https://llmfit.io/

https://modelfit.io/

At 8-bit quantization (q8_0) I get 20 tokens per second on a Radeon R9700.

Check out https://www.canirun.ai/ (and https://news.ycombinator.com/item?id=47363754)

Fwiw, huggingface does this on the page where you download the weights. Slightly different format though - you put all the hardware you have, and it shows which quants you can run.

Divide the value before the B by 2, and there's your answer if you get a Q4_K_M quant. Plus a bit of room for KV cache.

TLDR: If you have 14GB of VRAM, you can try out this model with a 4-bit quant.

Tokens per second is an unreasonable ask since every card is different, are you using GGUF or not, CUDA or ROCm or Vulkan or MLX, what optimizations are in your version of your inference software, flags are you running, etc.

Note that it's a dense model (the Qwen models have another value at the end of the MoE model names, e.g. A3B) so it will not run very well in RAM, whereas with a MoE model, you can spill over into RAM if you don't have enough VRAM, and still have reasonable performance.

Using these models requires some technical know-how, and there's no getting around that.

https://github.com/AlexsJones/llmfit gives a good approximation of that.

this is what the unsloth post is for

I just ran Unsloth's Q4_K_M quant on Macbook Pro M1 Max (2021), 32 GB RAM. Getting about 9 tok/s. Reported RAM usage 15.9 GB. It's okay, but a bit slow for my taste.

An example of a query and a response is below. It was done with 9.09 tok/s, 893 tokens, 7.56 seconds to first token, thought for 1 minute and 9 seconds.

Question: I gave the model this truth table and asked to tell me what boolean function would produce the output given A, B as inputs.

  | A | B | Output |
  | :--- | --- | ---: |
  | (T) | (F)  | (T) |
  | (T) | (T) | (F) |
  | (F) | (F) | (F) |
  | (F) | (T) | (F) |

  The boolean condition for your table is simply:

  ```rust
  A && !B
  ```

  ### Why?
  Your table shows the output is `true` **only** when `A` is true and `B` is  false. That's exactly what `A && !B` expresses.

  | A | B | `A && !B` | Output |
  |---|---|-----------|--------|
  |  (T) |  (F) | `T && T = T` |   |
  |  (T) |  (T) | `T && F = F` |   |
  |  (F) |  (F) | `F && T = F` |   |
  |  (F) |  (T) | `F && F = F` |   |

  ### In Rust:
  ```rust
  if A && !B {
      // output is true
  } else {
      // output is false
  }
  ```

  This is the most direct and idiomatic way to express that logic. Let me know if you need it adapted back to your `live_ticks` / `on_disconnect` context!

depends on format, compute type, quantization and kv cache size.

27B dense is not a screamer, even on an RTX 6000, but it will run at full precision with (more than) enough room for context at the model's own capacity. You can expect about 30 tokens/second after prompt processing. Quants will likely run similarly well on the 16/24/32 GB consumer GPUs.

The 3.5 27B model was a strong and capable reasoner, so I have high hopes for this one. Thanks to the team at Qwen for keeping competition in this space alive.

I would detest the time/words it takes to hand hold through such a review, of teaching folks the basics about LLM like this.

It's also a section that, with hope, becomes obsolete sometime semi soon-ish.

[dead]