The persistent KV cache is interesting; I’d love to see how much of the speedup remains when requests share less structure.