Skip to content

Skip reactions to Telegram updates that arrive more than 5 minutes late - #306

Merged
iamwavecut merged 1 commit into
mainfrom
fix/drop-stale-telegram-updates
Nov 20, 2026
Merged

iamwavecut merged 1 commit into
mainfrom
fix/drop-stale-telegram-updates

Conversation

@iamwavecut

Copy link
Copy Markdown
Owner

Problem

On 2026-11-20 the Bot API stopped serving this bot from 13:42 to about 15:04 UTC. When it recovered, Telegram redelivered the backlog: from 15:00 to 15:36 UTC the inbox received 88,146 messages and edits, and 57,866 of them were more than 5 minutes old (the oldest by 4.5 hours). The Postgres inbox consumer deliberately ignored Telegram event time, so the bot answered hour-old chat, drew old requests and queued old voice messages while live messages waited.

Change

  • openplotva-updates: update_event_unix_date reads the Telegram time of the user action (date of a message, edit_date of an edit). update_is_late_at says whether that time is older than a limit.
  • stale_update_requires_handle now also keeps refunds and the service messages that change who is in a chat or its id (new or left members, group to supergroup migration, group created). Successful payments were already on the list.
  • openplotva-app inbox consumer: a message or edit older than 5 minutes at claim time goes through UpdateHandler::record_late_update. UpdateHandlerWithHistory writes it to history and does not call the router. The inbox row closes as ignored with reason stale_update, so the skip can be counted in SQL and the runtime API.
  • Callback, inline and pre-checkout queries carry no action time and are handled as before. Bot and member status updates are applied by the materializer and are not affected.

Verification

  • cargo fmt --all
  • cargo clippy --workspace --all-targets -- -D warnings on the pinned 1.99.0 toolchain
  • cargo test -p openplotva-updates -p openplotva-app: 93 and 1473 passed
  • New tests: event-time rules (old and fresh message, edit by edit_date, button under an old message), the exemptions (payment, refund, new members), history without routing, and the inbox outcome mapping

After the 2026-11-20 Bot API outage Telegram redelivered about 58,000
messages that were up to 4.5 hours old. The Postgres inbox consumer
handled every one of them, so the bot answered hour-old chat, drew old
requests and queued old voice messages, while live traffic waited.

The consumer now compares the Telegram time of the user action with
the claim time. A message or edit older than 5 minutes is written to
history and closed as ignored (`stale_update`) without reaching the
router. Payments, refunds and membership or chat-id service messages
still run their handlers however late they are. Callback, inline and
pre-checkout queries carry no action time and are handled as before.
@github-actions

Copy link
Copy Markdown
Messages
📖 Danger checked 2 changed file(s).

Generated by 🚫 dangerJS against e60fd54

@iamwavecut
iamwavecut merged commit c0f826b into main Nov 20, 2026
12 checks passed
@iamwavecut
iamwavecut deleted the fix/drop-stale-telegram-updates branch November 20, 2026 15:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant