logoalt Hacker News

jLaForesttoday at 11:34 AM1 replyview on HN

I'm in the middle of moving my app to openAI structured output.

Could you please explain what you mean by "which everyone moved on from"?


Replies

slickytailtoday at 11:47 AM

Structured decoding limits next-token probabilities to ensure valid JSON. The main issue is that if the model puts a substantial probability on an invalid token, then it was already confused, and in that case, you don't actually want whatever the next-most-likely valid token is: even if it's syntactically valid, it's likely semantically erroneous.

show 2 replies