alt Hacker NewsI have a 16GB 5060 and 32GB system. RAM - are there any recommended Qwen 3.8 Flash models worth running on this ? I am using Nemotron 30B A3B at the moment
Replies
MaxikCZ • today at 11:20 AM
32 gb of ram is gonna be your blocker for iq3, u will have to go iq2 and thus the output will suffer. At the plus side, it will spew faster than iq3.
Iq3xxs fits barely on my 10+64gb, so much that I am agresivelly cashing checkpoints to disk instead of keeping them in memory, moved vision from ram to disk too etc, I am sitting at few gigs left RAM and few hundred MB free on gpu.
Fwiw I think you can run 3.8 flash (next?) on your setup with SSD streaming.
There's this video (1) that I found, and my understanding is that flash next can be very efficient wrt ram needed, by using a "dictionary" (or something similar).
1 - https://youtu.be/IH8XmxiwliQ?si=lJPvYzi3mHNsiw8r