alt Hacker NewsI have access to a server with 4 x h200 cards - 540 GB in total memory, but it appears I have to go to 4bit or 3bit even to serve it. Any idea if that is feasible/desirable (we have restrictions that we can't use a chinese model!).
Replies
segmondy • today at 3:45 PM
4 bit quality would be good, but you need more GPU, the 4bit weight is 610gb. EXL3 in 3 bit has been okay the few times I used it. https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B-NVF...
himata4113 • today at 2:53 PM
use MoE offloading
echelon • today at 2:53 PM
> we have restrictions that we can't use a chinese model!
Please expand on this if you can. Why can't you? Who mandated this?
➕ show 2 replies
One thing you should have figured out by now is that you can't trust what other people say because they know just about as much as you do and there's good odds they did it wrong.
4 bit quants can be fine, try it and see.