These benchmarks and rankings seem more noise than signal these days. They're uniformly divorced from sincerely-attested real world experience. And the real world experience isn't driving any consensus either.

So here's a meta-benchmark: at what point does the act of choosing an LLM product for a reasoning task become less reliable (in the sense of "predictive of future success") than hiring a human being for that same task? Have we already cross that threshold? I think we might have.

And once we do... what happens then? Will the truly elite models be hidden in a sea of mediocrities simply because the goofball "managers" doing the hiring are too dumb to recognize their genius?