logoalt Hacker News

jacquesm • yesterday at 9:28 PM • 3 replies • view on HN

I always wonder what you lose when you start maxing out some specific benchmark. It is really hard to do this in a balanced way and I would love if everybody that did these kind of efforts would take the time to also list the before/afters of a number of other benchmarks that they did not use in their training.


Replies

nblintao • yesterday at 9:46 PM

Good point! First of all, none of the benchmark tasks were used in training, though the training data was designed to improve capabilities on tasks like TB4.1. If you consider TB4.1 the main target, the blog actually shows before/after results on several other benchmarks, with no regressions and even some improvements.

➕ show 1 reply
gardnr • yesterday at 9:42 PM

Check the article. It’s the first graphic that has other benchmarks.

➕ show 1 reply