logoalt Hacker News

Manabu-eoyesterday at 6:15 PM5 repliesview on HN

How likely this problem is already on the training set by now?


Replies

simonwyesterday at 6:24 PM

If anyone trains a model on https://simonwillison.net/tags/pelican-riding-a-bicycle/ they're going to get some VERY weird looking pelicans.

show 1 reply
throwup238yesterday at 6:20 PM

For every combination of animal and vehicle? Very unlikely.

The beauty of this benchmark is that it takes all of two seconds to come up with your own unique one. A seahorse on a unicycle. A platypus flying a glider. A man’o’war piloting a Portuguese man of war. Whatever you want.

show 1 reply
zarzavatyesterday at 6:22 PM

You can always ask for a tyrannosaurus driving a tank.

verdvermyesterday at 6:18 PM

I've heard it posited that the reason the frontier companies are frontier is because they have custom data and evals. This is what I would do too