Skip to content

fix(ci): pre-fetch Qwen tokenizer vocab with retries to deflake tests - #242

Merged
lzsweb merged 1 commit into
mainfrom
fix/ci-tokenizer-vocab-flaky
Nov 20, 2026
Merged

lzsweb merged 1 commit into
mainfrom
fix/ci-tokenizer-vocab-flaky

Conversation

@lzsweb

@lzsweb lzsweb commented Nov 20, 2026

Copy link
Copy Markdown
Collaborator

Summary

  • test_root_get_tokenizer_encodes_locally (and the legacy tokenizer tests) fail intermittently because the first-use vocab download from ModelScope times out on GitHub runners (e.g. run 44684550767, Python 3.10 leg: The read operation timed out).
  • Both test jobs now cache ~/.cache/dashscope/tokenizers across runs and pre-download the vocab with 5 retries and backoff before pytest, reusing the SDK's own URL/SHA256/cache logic (_resolve_vocab_path). Cache hits skip the network entirely.
  • CI-only change; no SDK code or test semantics modified.

Test plan

  • Workflow YAML parses; pre-download snippet runs locally (cache-hit path)
  • tests/test_package_root.py::test_root_get_tokenizer_encodes_locally + tests/legacy_1x/unit/test_tokenizer.py pass locally (12 passed)
  • Unit Tests matrix green on this PR, including the previously flaky 3.10 leg

The vocab download from ModelScope times out intermittently on GitHub
runners, failing test_root_get_tokenizer_encodes_locally and the legacy
tokenizer tests. Cache the vocab across runs and pre-download it with
retries in both test jobs, reusing the SDK's URL/SHA256/cache logic.
@lzsweb
lzsweb merged commit a23c72c into main Nov 20, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant