Skip to content

Bound Telegram file lookups and media sends in queue workers - #305

Merged
iamwavecut merged 1 commit into
mainfrom
fix/telegram-file-wait-bounds
Nov 20, 2026
Merged

iamwavecut merged 1 commit into
mainfrom
fix/telegram-file-wait-bounds

Conversation

@iamwavecut

Copy link
Copy Markdown
Owner

Problem

On 2026-11-20 from 12:03 UTC Telegram stopped answering getFile and media uploads for the bot, while getMe and text sends kept working. Every such call waited for the shared 90 s HTTP timeout, which exists for the getUpdates long poll.

  • Each ASR job took the accelerator slot, waited 90 s on getFile and failed. ASR outranks drawings in accelerator arbitration, so VIP drawings started only after priority aging (336 s and 645 s) and regular ones after 30 minutes.
  • Each finished picture then spent 90 s per upload. Three serial timeouts kept the slot for 270 s more and the job failed with image generation produced no image.
  • The upload error was logged at debug level, so production logs showed nothing.

Change

  • openplotva-telegram gets get_telegram_file, get_telegram_file_path and download_telegram_file. getFile gives up after 10 s, a download after 30 s. The bound also covers the client's own 429 retry sleeps.
  • The six copies of the lookup and download loop use the helper: ASR, vision, music references, image edit references, the image edit file probe and the Python sandbox. Size limits and their error texts stay the same.
  • Telegram calls made by the image and video workers (placeholders, picture uploads, sendAnimation) give up after 30 s.
  • A failed placeholder fill is logged at warn level.

Not changed: the shared 90 s client timeout (the long poll needs it), the dispatcher send path, and the 120 s wait of a dialog turn for its ASR job (it already returns as soon as the job stops being active).

Verification

  • cargo fmt --all -- --check
  • cargo clippy --workspace --all-targets -- -D warnings on the pinned 1.99.0 toolchain
  • cargo test -p openplotva-telegram -p openplotva-app: 241 and 1468 passed
  • New tests: getFile against a server that never answers, a download whose body stalls, the size limit, and the 30 s worker bound under paused time

On 2026-11-20 Telegram stopped answering getFile and media uploads for
the bot. Every call waited for the shared 90 s HTTP timeout, which is
sized for the getUpdates long poll. Each ASR job held the accelerator
slot for 90 s before failing, so drawings started only after priority
aging, and each finished picture spent up to 270 s in three serial
upload timeouts before the job failed.

- getFile now gives up after 10 s and a file download after 30 s. One
  helper in openplotva-telegram replaces six copies of the lookup and
  download loop (ASR, vision, music references, image edit references,
  the image edit file probe, the Python sandbox).
- Telegram calls made by the image and video workers give up after
  30 s.
- A failed placeholder fill is logged at warn level. It was logged at
  debug level, so the delivery failures left no trace in production.
@github-actions

Copy link
Copy Markdown
Messages
📖 Danger checked 12 changed file(s).

Generated by 🚫 dangerJS against 02fe574

Comment thread crates/openplotva-app/src/image_jobs.rs Resolved
@iamwavecut
iamwavecut merged commit d91f310 into main Nov 20, 2026
12 checks passed
@iamwavecut
iamwavecut deleted the fix/telegram-file-wait-bounds branch November 20, 2026 13:02
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant