alt Hacker NewsIf a given paragraph gets encoded into twice as many tokens, that means the model gets twice as many matmuls to process it. The amount of compute thrown at the problem is increased (everything else constant), which may improve the quality of the result. This is believed to be one of the reasons that 'thinking' tokens improve quality. For long tasks it will lead to more context compactions though which will harm the quality to some degree as well.
It would be nice if inference could somehow perform token generation using "contractions" of "fluffy" tokens, where combining those tokens doesn't decrease nuance but provides additional efficiency. That may already be happening - I haven't looked at the most modern methods of inference in a long long time.