logoalt Hacker News

p0w3n3dtoday at 7:07 AM1 replyview on HN

I think the gain is very little. Almost every English word is on token, the same with programming language keywords. So you're just replacing one keyword with another. The only gain in the example given is > instead of jsonify() which would be ~4 tokens.

Please check your idea agains tiktokenizer


Replies

p0w3n3dtoday at 2:23 PM

I've checked and you get 36->30 tokens decreasal but no human readability. sounds like a poor trade