logoalt Hacker News

Perplexity and Intelligence

39 points • by acossta • today at 5:06 PM • 8 comments • view on HN

Comments

jrflo • today at 5:47 PM

This trend is pretty shocking to me. I assumed that we had moved beyond cross-entropy loss being important with the increasing use of RL, it's wild that the trend is so clear. Really nice to have such a simple benchmark instead of the increasingly convoluted ones that are getting spun up every day.

Also, I wonder what happened with GPT 4.5? I think o1 was the start of true reasoning models, right? I was using Claude at the time so I can't remember.

➕ show 1 reply
amluto • today at 5:44 PM

> The primary held-out datasets we’ve been using since GPT-3.5 are sourced from our internal codebase, and those are the numbers reported here (for a fixed snapshot).

Why would OpenAI expect any meaningful relationship between their models’ ability to predict the next token of supposedly held-out chunks of their internal codebase and the overall capabilities of the models? I expect both to improve, but that’s about the extent of it.

Leary • today at 5:35 PM

Looks like their latest internal model performs somewhat similarly to model 4 of anthropic.

eulgro • today at 5:45 PM

Didn't they hit a data wall roughly two years ago and had to rely on artifical datasets for pretraining i.e. LLM-generated?

I would expect the entropy of artificial datasets to be lower, thus easier to predict and scoring better on "perplexity"?

➕ show 1 reply
Retro_Dev • today at 5:45 PM

Saying that perplexity is a metric of intelligence is misleading. Inkling-2-Mini was released recently, and I've measured its perplexity at 5.7 on a small varied text dataset. For reference, the graph OpenAI is glazing is plotting log(PPX), which would put this model at a 0.756 on the Y Axis... Which is apparently between GPT 6.2 Astra and their internal model. Perplexity can also give the same number for a model which understands text as to one that memorizes text... And as I'm pretty sure Inkling-2-Mini is smaller than GPT 6.2 Astra - and it's generally acknowledged that number of parameters equals memorized knowledge, this shows how weak this advertising stunt for OpenAI is.

➕ show 2 replies
rfgplk • today at 5:10 PM

What's the perplexity of the average person?

➕ show 2 replies