alt Hacker NewsWhat I really want to know for models like this is if I can have them run on really lame consumer hardware (consider 128gb of ram, 2080ti) with extremely low TPS, but with reasonable enough intelligence that you can walk away and expect something to bake really nicely while you sleep.
Perhaps I'm losing touch with the edge, but it doesn't seem like this is quite possible with the hardware baseline I'm talking about...
Replies
I use DwarfStar DeekSeek V4 Flash on my Mac Studio with 128gb of RAM and it's pretty good - decent TPS and coding ability. Apparently the V4.1 model can also be run by streaming weights from the SSD - I've not tried this yet because it results in a lower TPS, but eventually this version of 4.1 should also be usable like this.
expert streaming with cache-conditional routing is very powerful (you can run 4.1 Flash on two Strix Halos and a 9070XT at around 50-70TPS decode in ~full quality), and that setup might be able to at least do a few tok/s with it.
You sure can, https://github.com/lxe/dsx gets you this deepseek on 2x3090s and 128 gb ram at about 30 t/s
At around 75% of the full model intelligence
You can get something relatively capable and usable with your configuration and Strata: https://github.com/Niko1221/Strata
Nvidia stopped manufacturing the 5090, so you and people like us cannot run these models on our computers and are forced to use American SOTA so they don't lose the game...
[dead]
"Model like this" not sute about ds v4.1 but qwen3.8 flash next is currently running on 10gb 3080 + 64 gb ddr5 ram at respectable 40-70t/s with 1200pp at max context