alt Hacker NewsI'm working on TokenDagger [0] a high performance implementation of OpenAI's Tiktoken. My benchmarks are showing 2-3x higher throughput, as well as ~4x faster tokenization for code samples on a single thread.
I'm working on TokenDagger [0] a high performance implementation of OpenAI's Tiktoken. My benchmarks are showing 2-3x higher throughput, as well as ~4x faster tokenization for code samples on a single thread.