logoalt Hacker News

kennywinkertoday at 4:28 PM1 replyview on HN

Total param count decides how much vram you need to run it. Active param count decides how fast it runs. My 10 year old GPU can load quantized 35B or 27B, but it can’t process 27B parameters per token faster than 2-4tok/s, while it can do A3B at >40tok/s


Replies

Alifatisktoday at 4:53 PM

Thank you Kenny