alt Hacker NewsAm I reading the Agents’ Last Exam v1 graph correctly where it's saying GPT Luna Medium performs better than High, but XHigh is better than Medium?
Replies
ttul • today at 8:05 PM
That is correct. The gpt-6 models are strange.
Am I reading the Agents’ Last Exam v1 graph correctly where it's saying GPT Luna Medium performs better than High, but XHigh is better than Medium?
That is correct. The gpt-6 models are strange.
The error bars on eval graphs are commonly like ±5%. It's fairly common to get out-of-order points when the gap between them is only a few percentage points. If we ran it again, or ran it with minor alterations, I'd expect the ordering to be expected. In general, we don't massage any of our eval numbers and for good statistical hygiene we'll report the first run we do, even if later runs show slightly smoother curves.
(I work at OpenAI.)