alt Hacker NewsBecause people, for some stupid reasons, still read and make decisions by these public benchmarks, which makes zero sense. Create your own benchmarks and evaluations, then use those, it's pretty the only way to get any qualitative answers to how well a model will work for you and your user cases.
How many users are willing to spend the time and repeated cost (once for each new potential model+version+variant) for that - 1 in hundreds? (Edit: I was thinking about this as an individual dev. Especially for AI uses in a biz workflow, custom benchmarking makes perfect sense yes.)