alt Hacker Newstl,dr: They used a rubric to have the LLM grade the chunks on a Likert scale. I think this is a good way to coax numbers out of an LLM.
tl,dr: They used a rubric to have the LLM grade the chunks on a Likert scale. I think this is a good way to coax numbers out of an LLM.
Likert scale just doesn't work in LLM evals. My idea of 3/5 is different from your 3 and definitely different from an non-deterministic system's 3.