alt Hacker Newsbtw that how mmlu score things to answer question instead of producing all the answer token they look at logprob of a b c d keys in 2020 making this technique old as dirt in nlp
btw that how mmlu score things to answer question instead of producing all the answer token they look at logprob of a b c d keys in 2020 making this technique old as dirt in nlp
This technique is so obvious to anyone who spends more than a minute with multiple choice tasks. It's wild they're claiming it as a feature.