Repository navigation
Keep a server that still answers, and never lose what it already delivered - #38
Open
SerhiyGreench wants to merge 1 commit into
Open
SerhiyGreench wants to merge 1 commit into
SerhiyGreench wants to merge 1 commit into
Conversation
…vered A cluster manager removed a server when its announcements lapsed for the alive timeout, and the clustered queue tore its connections down at once: destroy() removed the message listeners, then dropped the reader. A reply the reader had already taken from the broker but not yet read was lost, and its caller waited out its whole timeout. A busy process misses announcements as easily as a dead broker stops sending them, so a production gateway lost calls this way under load while its broker was healthy. - UDPWorker says whether a removal is a lapse or a down notice, and the UDPClusterManager hands that to the cluster as ServerRemoval.lapsed. - ClusteredRedisQueue PINGs a server whose announcements only lapsed, and keeps it where it answers within IMQ_LAPSED_ANSWER_TIMEOUT (2 s). - A server that is removed is drained first: RedisQueue.drain() releases the blocked reader with CLIENT UNBLOCK from the writer, so a reply already in its socket reaches the listeners before teardown; nothing further is taken. - RedisQueue.answers() says whether the broker answers a PING in time. Unit specs cover both, and test/integration/removedServer.spec.ts proves them against a real broker: a reply in the reader's socket while the event loop is held is delivered across the removal (lost on master), a lapsed broker that answers is kept, and a drained reader takes nothing more.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A cluster manager removed a server when its UDP announcements lapsed, and the clustered queue tore its connections down at once: destroy() removed the message listeners, then dropped the reader. A reply the reader had already taken from the broker, but not yet read, was lost, and its caller waited out its whole timeout. A busy process misses announcements as easily as a dead broker stops sending them; a production gateway lost 16 calls this way in one test run while its broker was healthy.
What changes
ServerRemoval.lapsed(a new optional second argument ofICluster.remove).IMQ_LAPSED_ANSWER_TIMEOUT(2 s). A down notice still removes at once.RedisQueue.drain()releases the blocked reader withCLIENT UNBLOCKsent on the writer, so a reply already in its socket reaches the listeners before teardown. Nothing further is taken.RedisQueue.answers()says whether the broker answers a PING in time.Tests
test/integration/removedServer.spec.tsagainst a real broker: a reply in the reader's socket while the event loop is held is delivered across the removal; a lapsed broker that answers is kept; a drained reader takes nothing more. All three fail on master (the first loses the reply), and pass here, beside the existing clusterSubscription specs.