alt Hacker News4 bit quality would be good, but you need more GPU, the 4bit weight is 610gb. EXL3 in 3 bit has been okay the few times I used it. https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B-NVF...
4 bit quality would be good, but you need more GPU, the 4bit weight is 610gb. EXL3 in 3 bit has been okay the few times I used it. https://huggingface.co/mistralai/Mistral-Large-4-1T-A52B-NVF...