alt Hacker NewsHow does this compare to the BPE crate [1]? Its main selling point is support for incrementally re-tokenising text, but it's also faster than tiktoken.
How does this compare to the BPE crate [1]? Its main selling point is support for incrementally re-tokenising text, but it's also faster than tiktoken.
I'm working on incremental re-tokenizing next. Then I'll run some benchmarks against this crate too.