Wouldn't caching be in favor of lowest possible latency?
It might be faster to burn the compute instead of having to fetch the KV-cache over the network from an SSD.
alt Hacker News
It might be faster to burn the compute instead of having to fetch the KV-cache over the network from an SSD.