Skip to content

ci: pre-download Qwen tokenizer vocab to deflake tests - #297

Open
lzsweb wants to merge 1 commit into
mainfrom
ci/pre-download-qwen-vocab
Open

lzsweb wants to merge 1 commit into
mainfrom
ci/pre-download-qwen-vocab

Conversation

@lzsweb

@lzsweb lzsweb commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • Since refactor(tokenizers): stop shipping qwen.tiktoken in jars #293 the qwen.tiktoken vocab is downloaded from ModelScope on first use; tokenizer tests (TestQwenTokenizerVocabResolution, TestTikTokenTokenizer) can hit the network during mvn test, which is flaky from overseas GitHub runners (same issue as the Python SDK's test_root_get_tokenizer_encodes_locally).
  • Add a pre-download step to both build-and-test.yml and release.yml: fetch the vocab into ~/.cache/dashscope/tokenizers/ with curl retries (3 outer attempts × 3 curl retries) and SHA256 verification against the constant in QwenTokenizer.
  • QwenTokenizer.resolveVocab() honors the verified cache before downloading, so tests run fully offline once the step succeeds; no production code or test changes needed.

Test plan

  • Verified download URL and SHA256 (b2b1b8df...) locally
  • mvn test -Dtest='TestQwenTokenizerVocabResolution,TestTikTokenTokenizer': 11 tests, 0 failures
  • CI runs on this PR exercise the new step

The vocab is fetched from ModelScope on first use, which is flaky from
overseas runners. Pre-download it into the SHA256-verified local cache
(with retries) in both the test and release workflows so tokenizer
tests never hit the network during mvn test.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant