Do you have a ballpark idea of how much RAM would be necessary to run llama 3.1 8b and 70b on 8-quant?
Roughly, at Q8 the model sizes translate to GB, so ~3 and ~70GB.
alt Hacker News
Roughly, at Q8 the model sizes translate to GB, so ~3 and ~70GB.