Skip to content

Keep a server that still answers, and never lose what it already delivered - #38

Open
SerhiyGreench wants to merge 1 commit into
masterfrom
keep-a-healthy-broker
Open

SerhiyGreench wants to merge 1 commit into
masterfrom
keep-a-healthy-broker

Conversation

@SerhiyGreench

Copy link
Copy Markdown
Member

A cluster manager removed a server when its UDP announcements lapsed, and the clustered queue tore its connections down at once: destroy() removed the message listeners, then dropped the reader. A reply the reader had already taken from the broker, but not yet read, was lost, and its caller waited out its whole timeout. A busy process misses announcements as easily as a dead broker stops sending them; a production gateway lost 16 calls this way in one test run while its broker was healthy.

What changes

  • UDPWorker says whether a removal is a lapse or a down notice; UDPClusterManager hands it to the cluster as ServerRemoval.lapsed (a new optional second argument of ICluster.remove).
  • ClusteredRedisQueue PINGs a server whose announcements only lapsed and keeps it if it answers within IMQ_LAPSED_ANSWER_TIMEOUT (2 s). A down notice still removes at once.
  • A removed server is drained first: RedisQueue.drain() releases the blocked reader with CLIENT UNBLOCK sent on the writer, so a reply already in its socket reaches the listeners before teardown. Nothing further is taken.
  • RedisQueue.answers() says whether the broker answers a PING in time.

Tests

  • Unit specs for both paths (the mock learns PING and CLIENT ID/UNBLOCK).
  • test/integration/removedServer.spec.ts against a real broker: a reply in the reader's socket while the event loop is held is delivered across the removal; a lapsed broker that answers is kept; a drained reader takes nothing more. All three fail on master (the first loses the reply), and pass here, beside the existing clusterSubscription specs.

…vered

A cluster manager removed a server when its announcements lapsed for the
alive timeout, and the clustered queue tore its connections down at once:
destroy() removed the message listeners, then dropped the reader. A reply
the reader had already taken from the broker but not yet read was lost, and
its caller waited out its whole timeout. A busy process misses announcements
as easily as a dead broker stops sending them, so a production gateway lost
calls this way under load while its broker was healthy.

- UDPWorker says whether a removal is a lapse or a down notice, and the
  UDPClusterManager hands that to the cluster as ServerRemoval.lapsed.
- ClusteredRedisQueue PINGs a server whose announcements only lapsed, and
  keeps it where it answers within IMQ_LAPSED_ANSWER_TIMEOUT (2 s).
- A server that is removed is drained first: RedisQueue.drain() releases the
  blocked reader with CLIENT UNBLOCK from the writer, so a reply already in
  its socket reaches the listeners before teardown; nothing further is taken.
- RedisQueue.answers() says whether the broker answers a PING in time.

Unit specs cover both, and test/integration/removedServer.spec.ts proves
them against a real broker: a reply in the reader's socket while the event
loop is held is delivered across the removal (lost on master), a lapsed
broker that answers is kept, and a drained reader takes nothing more.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant