alt Hacker NewsAs an example, 2026 GPT doesn't even agree with its 2025 self. Last year I asked it to make a hardware comparison and it correctly identified the objectively better option. Recently I asked again and this time and it got everything completely backwards.
Models are stochastic. Did you look at pass@k? I wouldn’t be surprised if you saw a regression because these models are extremely complex and impact of various decision making downstream is complex.