Active parameter count really drives inference speed not ram requirements.
For this model, assuming you want the full 1mm context length at 4-bit quants, you need something like 175 GB total VRAM.
~150GB of that will be weights (varies a bit by quant format) then ~3-5GB for kv cache (might be lower here, I think they're using mostly HCA compressed cache) and then I'll allocate a couple gb for overhead/concurrency (say another ~10GB as a safety buffer).
---
So the good news is that it comfortably fits into 2x128 at 4-bit, and we can realistically stretch to a 6-bit quant and make it just fit (~244GB needed for the full 1mm context).
alt Hacker News
Mostly yes, you are wrong.
Active parameter count really drives inference speed not ram requirements.
For this model, assuming you want the full 1mm context length at 4-bit quants, you need something like 175 GB total VRAM.
~150GB of that will be weights (varies a bit by quant format) then ~3-5GB for kv cache (might be lower here, I think they're using mostly HCA compressed cache) and then I'll allocate a couple gb for overhead/concurrency (say another ~10GB as a safety buffer).
---
So the good news is that it comfortably fits into 2x128 at 4-bit, and we can realistically stretch to a 6-bit quant and make it just fit (~244GB needed for the full 1mm context).