I thought they got like a logit mask that removes illegal next tokens, at least in llama.cpp and GBNF.
You're probably right about that. It looks like llama.cpp lets you specify a grammar file or a JSON schema - is that what you mean? If the syntax of your DSL can be expressed that way, I suppose you wouldn't have to validate it externally and retry.
alt Hacker News
You're probably right about that. It looks like llama.cpp lets you specify a grammar file or a JSON schema - is that what you mean? If the syntax of your DSL can be expressed that way, I suppose you wouldn't have to validate it externally and retry.