alt Hacker NewsCounterargument: this works for quick prototyping, but for any serious business, you will eventually develop a benchmark/eval to track how well the general model is working, and once you have that dataset, you might as well train a specific model
Replies
ACCount39 • today at 6:15 PM
Or not. And replace the generalist with the next generalists that gets you +15% on that benchmark for the same price, or gives you the same benchmark performance for half the price.
One advantage of using generalist models is that the generalists are improving - regardless of whether you're doing anything about it.
Jev's bet is that if it works well enough for random use cases that nobody complains, then management won't feel a need to develop a benchmark/eval, and they won't need to employ all those data science guys.