logoalt Hacker News

wincytoday at 4:40 AM0 repliesview on HN

With Ninfer and Qwen 3.8 27b it uses a groupwise int mixed quant, and it gets 160 tokens/sec. The mixed quant is between 4 and 6 bits.