alt Hacker NewsHow large?
With 64 GB of unified memory, you should be able to run a DeepSeek V4 Flash quantisation at 7–10 t/s, for example with: https://github.com/antirez/ds4 or https://github.com/steadfastgaze/MoEspresso (my engine).
The routed experts needed for the next tokens that are not already in memory need to be read from the SSD, so the speed becomes SSD reading bound and the larger the memory, the faster the inference.
Replies
mandeepj • today at 5:14 PM
> 7–10 t/s
Maybe use it for overnight batch work! Hopefully, you aren’t suggesting it using for realtime conversations!
Is there a particular quant of DS v4 Flash you'd recommend that works on 64GB machines? None of the antirez versions on HF look small enough?
Also, FWIW, I've been experimenting with Laguna-S-2.1. It runs reasonably quickly (llama.cpp, IQ2_M quant) but the outputs so far aren't impressive, and it gets stuck and perseverates. Very subjectively, at that level of quantisation, it seems to perform worse than Qwen 3.6 27B at Q4_K_XL.