logoalt Hacker News

syntaxingtoday at 2:01 PM0 repliesview on HN

Which quants? I get these speed (only 20-30 TPS) at Q4_K_M with MTP for 27B on my framework desktop. I draw sub 130W for the whole machine.