This has to be a distilled model? I don't see any route to 200 tokens/s without serious datacenter hardware
you mean quantized? it’s only got 16B active parameters so it can be pretty fast at decode.
alt Hacker News
you mean quantized? it’s only got 16B active parameters so it can be pretty fast at decode.