Is that the case?
If the entire model fits in vram, won't the tps be comparable?
autoregressive generation is memory bandwidth bounded
alt Hacker News
autoregressive generation is memory bandwidth bounded