alt Hacker NewsInteresting... Naively I'd assume you'd have a pretty unfair advantage on quality if you have materially more information dense tokens.
That doesn't really appear to be the case as GPT and Anthropic models appear evenly matched despite Anthropic encoding the same text into almost ~2x the tokens...
I'd also - naively - assume this would make training their models more expensive. Though inference now dominates, and they'd probably rather have more tokens than less (to charge you for them at future 80% margins).
Replies
not-a-llm • yesterday at 7:49 PM
more dense tokens means more stuff to fit into the embedding space which is per token, so more work to disentangle later
If a given paragraph gets encoded into twice as many tokens, that means the model gets twice as many matmuls to process it. The amount of compute thrown at the problem is increased (everything else constant), which may improve the quality of the result. This is believed to be one of the reasons that 'thinking' tokens improve quality. For long tasks it will lead to more context compactions though which will harm the quality to some degree as well.