logoalt Hacker News

zenopraxtoday at 1:44 AM0 repliesview on HN

For some reason the unsloth models leave hardly any room for context. I've switched to the regular (non-unsloth) and get about 25 t/s and get about 80,000 more context tokens for the same quant.