Ask HN: How are thinking efforts implemented?

Take a look at the harmony repo which specifies the internal OpenAI format - the effort level is specified in the context after the <|start|> tag - https://github.com/openai/harmony

Note that inference libs also have parsers that put hard limits on reasoning tokens with separate counters (similar to how you can put a limit on token generation per completion versus waiting for an <eos>). For that, take a look at vllm reasoning docs.

Different models do slight variants.

Usually it’s done in post training to enforce behavior based on prompt. Ie. System prompt with thinking:max or low or wtv.

Enforcement then goes via constrained decoding, checking for think token start and end with max lengths, or other variations

LLMs work by generating the most likely continuation to a prompt. But they can also generate multiple likely continuations. This create multiple branches which in turn can generate even more branches. The LLM can then evaluate the branches, prune the unpromising ones, and merge the best ones. More branches means more tokens, means more effort.

At a guess. May be associated with token length context window. Down selecting is consistent with warning message, forcing cutoff to context window. The technical term cache being a synonym. Increasing the headroom for more "thinking" should allow the implementation to access more resources without warning about the cache breaking.

they use multitoken prediction behind the scenes, that might interact with the CoT in a strange way. maybe for different thinking modes they have different MTP models? if so thats interesting

Usually it’s not a different model, it’s the same model with different inference-time settings. “Thinking effort” typically changes the compute budget and decoding behavior (how many steps, how much exploration, sometimes internal planning loops).

Some stacks also tie it to orchestration layers or system/prompt signals, which is why it can look inconsistent across products

[dead]