The persistent KV cache is interesting; I’d love to see how much of the speedup remains when requests share less structure.
Why qwen 2.5 everywhere? Why not 3.5?
alt Hacker News
The persistent KV cache is interesting; I’d love to see how much of the speedup remains when requests share less structure.