logoalt Hacker News

chilmers • yesterday at 6:31 PM • 2 replies • view on HN

Here's a good heuristic for predicting the ability of a modern frontier LLM at any non-automatable task: Is it hugely overrepresented in their pre-training dataset (e.g. code), and were they specifically post-trained on it using reinforcement learning (e.g. coding)? If the answer to one or both is yes, you should expect them to be very good at it. Otherwise, you should expect them to be, at best, as good as an average human who's read a few books on the subject and is good at bullshitting.


Replies

boelboel • yesterday at 7:16 PM

"average human who's read a few books on the subject and is good at bullshitting" seems like a large portion of white collar workers that make bank.

Creamsicle47 • yesterday at 7:06 PM

The most sane comment I've seen in this thread