logoalt Hacker News

merbanan • today at 1:55 PM • 0 replies • view on HN

Q2_0 does 33 tok/s decode and ~600t/s prompt processing at 128k context on RTX2060 8GB VRAM.

ISTA IQ3_XXS does ~21 tok/s decode and ~240t/s prompt processing