LLM token probabilities are really poorly calibrated for decision making.

Every time I see someone saying something like "isn't this just structured output?", it's because they don't understand that probability calibration is one of the most important parts of the product, especially to understand uncertainty.

I think Jev did a pretty good job on calibration. I wonder how the fast-follow competition from OpenAI and Gemini do.