alt Hacker News>while having way less diversity of answers to that question than humans would.
Why is this ironic? The actual number of different LLMs is tiny. And a lot of those are just distilled from SOTA models.
>The weights are initialized randomly and in theory the labs should have quite some divergence in datasets by now due to different scraping,
I think the opposite, the models actually converge over time. They ingest FAR more data than any human ever could in countless lifetimes. Any one particular bit of data may only have a part in a trillion of effect. From this data you start to reach means and statistical averages across all models.
Now, I don't always believe it will be this way. As training models get cheaper and faster we will see unique LLMs start to diverge and be used more often.
The models in agreement here are all the frontier labs. I don't believe Google is distilling Anthropic or Anthropic is distilling OpenAI.
I don't think we're seeing averages or obvious fix points in the dataset. People went looking for the origins of Elias the Lighthouse Keeper and couldn't find one. Nobody predicted this outcome, it was discovered. It's a weirdly specific character and name choice, one that isn't obviously dictated by "the training set" (whatever that is), and yet they all reached the same place. How can such an obscure point in latent space be optimal for every model??