I'm sorry, I'm going to sound dumb for a second.

I know that you can use LLMs to write ascii numbers that represent a "confidence" as in TFA. This is fraught and strange and hard to trust.

But can't you also use the next token likelihood itself, so long as you rearchitect to use the input classes as the only possible output tokens? Something like stripping away speech and only working with the input labels as vocabulary...

I guess I thought that's what decision-like models were doing to output confidence scoring. (or something like that)