Skip to content

fix: reuse one tokenizer instead of building one per count - #679

Open
Tarquinen wants to merge 1 commit into
devfrom
fix/tokenizer-reuse
Open

Tarquinen wants to merge 1 commit into
devfrom
fix/tokenizer-reuse

Conversation

@Tarquinen

Copy link
Copy Markdown
Owner

Fixes #638

  • countTokens now builds the @anthropic-ai/tokenizer tokenizer once and reuses it, instead of calling the package's countTokens(), which builds and frees a new WASM tokenizer every call. Same NFKC + special-token handling, so counts are identical.
  • Slow since the switch from gpt-tokenizer in 9810711. 500 counts of 2.4 KB: ~11.5 s before, ~0.1 s after.

Tested: new unit test (counts match the package, 200 calls < 2 s; fails on dev at 4.6 s) + V2 sandbox on OpenCode 2.3.0 reading a 2 MB log in 6 chunks with 6 successful compressions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant