logoalt Hacker News

computomatic • yesterday at 8:28 PM • 1 reply • view on HN

I came across this recently and was initially surprised too. It makes sense when you consider that any company trying to build models (or an agent/value-add layer on top of a model) needs benchmarks. The best funded will do the full vertical - build their benchmarks and models together - but many will outsource the benchmark. This is particularly valuable when the dataset is difficult to compile (robotics being a prime example) - it makes more sense to let the benchmark vendor focus on that.


Replies

besterman23 • yesterday at 8:38 PM

Sure it’s useful, but I’m asking who actually pays for this. Sort of a conflict of interest if you’re providing public testing for the companies paying you for the tests no?

Perhaps it’s purely for VC so they can judge which model is best? Also seems a little too conflict of interest-y, but that might just be too cynical.