logoalt Hacker News

slickytailtoday at 11:47 AM2 repliesview on HN

Structured decoding limits next-token probabilities to ensure valid JSON. The main issue is that if the model puts a substantial probability on an invalid token, then it was already confused, and in that case, you don't actually want whatever the next-most-likely valid token is: even if it's syntactically valid, it's likely semantically erroneous.


Replies

rhodysurftoday at 1:34 PM

So whats the alternative? Letting it codegen a file and the piping that? What a worse workflow

slicestoday at 2:18 PM

sounds like an argument in favor of structured output, is that right?