alt Hacker NewsWhat is the purpose of this? Just a hard cutoff below the actual context window? You could set that in your harness anyway.
Replies
dnlzro • today at 8:34 PM
Uses less quota (i.e., cheaper). For people who like to keep their contexts small, this is a no-brainer.
surgical_fire • today at 8:33 PM
> k3 (1M) consumes about twice as much quota as k3-256k
Cheaper?
➕ show 1 reply
At least in the self-hosted LLM inference engines, you have to pre-allocate space for the maximum amount of context you want to allow for each parallel session. By using a lower maximum, you don't have to allocate as much VRAM for each session, allowing more usage for the same amount of hardware. Thus, cheaper.