logoalt Hacker News

freehorsetoday at 6:20 PM0 repliesview on HN

I have found that q8 kv quantisation is really bad in anything requiring some kind of literal retrieval even at relatively shorter contexts (5-10k). Eg it would mess up names, titles of papers etc that were deeper in the context.