Skip to content

refactor(tokenizers): stop shipping qwen.tiktoken in jars - #293

Merged
lzsweb merged 4 commits into
mainfrom
dev/issues-13
Nov 19, 2026
Merged

lzsweb merged 4 commits into
mainfrom
dev/issues-13

Conversation

@lzsweb

@lzsweb lzsweb commented Nov 16, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Port dev v2.1 dashscope-sdk-python#237 to the Java SDK: the 2.4MB qwen.tiktoken vocab no longer ships in distribution jars.
  • QwenTokenizer resolves the vocab at runtime, in order: DASHSCOPE_QWEN_VOCAB env var (with ~ expansion) → classpath resource (source checkouts) → SHA256-verified cache under ~/.cache/dashscope/tokenizers → ModelScope download (temp file + SHA256 check + atomic replace), with offline manual-download guidance on failure. JDK-only implementation, Java 8 compatible, no new dependencies.
  • maven-jar-plugin and maven-source-plugin exclude qwen.tiktoken; the file stays in src/main/resources so source checkouts and tests keep working offline.

Test plan

  • New TestQwenTokenizerVocabResolution (env override, missing file, ~ expansion, sha256, classpath fallback) and existing TestTikTokenTokenizer pass; full test suite green.
  • Verified built jar and sources jar contain no qwen.tiktoken entry.
  • End-to-end: ran against the distribution jar without the vocab; first use downloads from ModelScope, verifies SHA256 (b2b1b8df…3186), caches under ~/.cache/dashscope/tokenizers, and encodes the expected token ids.

zhansheng.lzs and others added 4 commits November 16, 2026 18:02
Resolve the vocab at runtime from DASHSCOPE_QWEN_VOCAB, the classpath,
or a SHA256-verified ModelScope download cached under
~/.cache/dashscope/tokenizers, mirroring dashscope-sdk-python#237.
… cache

- Remove qwen.tiktoken from git and .gitignore tracking due to large file size
- Update pom.xml to exclude qwen.tiktoken from distribution jars with clarifying comments
- Modify QwenTokenizer to resolve vocab by downloading from ModelScope on first use and cache locally
- Allow manual local copy in classpath for offline development and testing
- Add test to ensure verified local cache exists before environment variable override tests
- Update test cases for default resolution from cache, classpath, or download and environment variable override usage
@lzsweb
lzsweb merged commit 007f528 into main Nov 19, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant