logoalt Hacker News

ariwilson • today at 7:41 PM • 3 replies • view on HN

I at least didn't see this. I had a baseline evaluation from before Nov 3 on Opus 5.5 medium, so I re-ran it this morning.

Eval context: 30 bug fixes and features evaluated by tests the agent didn't see. Ran on Opus 5.5 at medium effort in Claude Code.

20 of the tasks were last run on Nov 1, twice each. Nov 1 solved 36/40, and today solved 19/20. Thinking tokens per task were 0.95x Nov 1’s and output tokens 0.97x. The one significant change was speed: median time to first token decreased from 3.0s to 1.7s.

Caveats: one run per task today, medium effort only.


Replies

drakythe • today at 7:56 PM

If I read your results correct it solved 5% more problems and used up between 3-5% fewer tokens, with a faster response time?

➕ show 2 replies
risyachka • today at 8:47 PM

The spending limit definitely goes down much faster as of last week or so.