alt Hacker NewsMhh... my hunch is that part of this is that all python keywords are 1 token, I assume. And for those very weird languages, tokenizing might make it harder to reason over those tokens.
Would love to see how the benchmarks results change if the esoteric languages are changed a bit to make them have 1-token keywords only.
Considering that brainfuck only has 8 characters and models are scoring at 6.2% I don't think tokenization is the issue