alt Hacker NewsHow many users are willing to spend the time and repeated cost (once for each new potential model+version+variant) for that - 1 in hundreds? (Edit: I was thinking about this as an individual dev. Especially for AI uses in a biz workflow, custom benchmarking makes perfect sense yes.)
Probably the same amount of users who actually care about figuring out how well a model works for their use cases, instead of what amounts of guessing.
Most companies I've consulted for when it comes to LLM usage in various ways all have their own evaluations, not sure why others would avoid them.
This is to save time, for what it's worth. Doing that testing and evaluation will take you MUCH longer than doing so automatically, for obvious reasons.