alt Hacker NewsAt my org we've been building AI agents and one internal rule we have is to use at most 50% of the models context window with the recommendation to not go over 25% for large context window models.
Anytime I see a "1M Context Window", my brain always goes "Gotcha so a 250k usable window"
Replies
tedggh • today at 4:00 PM
Why 25% and not 12% or 40%? Is this an arbitrary vibestimate or you had some tests done that pointed you to 25%? I am genuinely interested in how others deal with context issues. Also I would think the usable context window is variable depending on the task, for example summarizing documents vs analyzing large, scattered and complex instructions.
➕ show 3 replies
Every single analysis I've seen done by anyone has basically come to the same conclusion: assuming sane BF16 or Q8 model quant, and BF16 or Q8 KV context quant (ie, not intentionally screwing over the model), you get about 250k before it pukes.