Repository navigation
fix(dwp): make federation lifecycle, reconnects and writes concurrency-safe - #57
Merged
Merged
Conversation
…y-safe Federation had several races that made Stop hang and could corrupt the peer stream. - Start overwrote f.cancel unconditionally, so a second Start left the first heartbeat and health loops running with no way to cancel them, and Stop then waited on them forever. Start is now a no-op while running. Stop is final: Start and AddPeer return ErrFederationStopped afterwards. - Stop waited on the WaitGroup before closing peer connections, but the group held peer read loops blocked in ReadServerText and reconnects sleeping on time.Sleep with context.Background. Stop now cancels a federation-owned context, closes every peer connection, and only then waits. Reconnects wait on a timer selected against that context, and connectPeer ties the dial and the auth exchange to it (closing the socket to interrupt the auth read), with a 30s handshake bound. Installing a connection and adding its read loop happen under the lifecycle lock, so nothing is added to the WaitGroup once Stop has begun and a connection cannot be installed after Stop or RemovePeer has retired the peer. - On a heartbeat timeout, checkPeerHealth started a reconnect and the read loop of the connection it closed started a second one; both dialled, leaking a connection and running two read loops on one conn. A per-peer reconnecting flag now allows a single reconnect at a time, the health check tears the connection down in the same critical section as its check, and each read loop owns only the conn it was started with, so a stale loop exits quietly instead of clobbering the new connection's state. - writeJSONFrame writes a WebSocket frame as a header and a payload with no lock, and heartbeats and forwarded requests write concurrently, so frames interleaved and the peer dropped the connection. Writes now go through a per-peer write mutex. - GenerateFrameID was a bare nanosecond timestamp. Clock resolution is coarser than that on some platforms, so concurrent requests shared an ID and one of them lost its response. IDs now carry a process-wide sequence number. - GetPeer and Peers handed out the live *Peer whose State and LastSeen are written under its lock. They now return copies taken under the lock; the type and signatures are unchanged. AddPeer with an existing ID now disconnects the peer it replaces, and a reconnect stops once its peer is removed or replaced. Tests use a fake WebSocket peer and cover Stop with a connected peer, in reconnect backoff and in a stalled reconnect handshake; double Start; Stop being final; a timeout producing exactly one reconnect; concurrent forwards arriving as well-formed frames; race-free peer snapshots; and unique frame IDs under concurrency.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
dwp.Federationhad races that madeStophang and could corrupt the stream to a peer. This PR fixes them and adds tests that drive a fake WebSocket peer.Bugs, triggers and fixes
1. Double
Startleaks loops. EachStartoverwrotef.cancel, so a second call left the first heartbeat and health loops running with nothing to cancel them.Stopthen waited on them forever.Fix:
Startis a no-op while running.Stopis final:StartandAddPeerreturnErrFederationStoppedafter it.Startnow returnserror; existing call sites still compile.2.
Stophangs with any connected or reconnecting peer.Stopcalledwg.Wait()before it closed peer connections. The WaitGroup held peer read loops blocked inReadServerTextand reconnects stuck intime.Sleepor a dial withcontext.Background.Fix:
Stopcancels a context the federation owns, closes every peer connection, and only then waits. Reconnects wait on a timer that also watches that context.connectPeerties the dial and the auth exchange to the same context (it closes the socket to interrupt the auth read), and the handshake is capped at 30s. The connection is installed and its read loop added under the lifecycle lock. So nothing joins the WaitGroup onceStophas started, and no connection is installed afterStoporRemovePeerhas retired the peer.3. A heartbeat timeout opens two connections.
checkPeerHealthclosed the connection and started a reconnect. The read loop of that closed connection started a second one. Both dialled, which leaked a connection and ran two read loops on one conn.Fix: a per-peer
reconnectingflag allows one reconnect at a time. The health check closes the connection inside the same critical section as its timeout check. Each read loop only acts on the connection it was started with, so an old loop exits quietly and doesn't change the new connection's state.4. Concurrent writes interleave frames.
writeJSONFramewrites a frame header and its payload as two separate writes with no lock. Heartbeats andForwardJob/ForwardEvent/Broadcast*write at the same time, so their bytes interleaved and the peer dropped the connection.Fix: all writes to a peer go through a per-peer write mutex.
5.
GetPeer/Peersreturned the live*Peer. ItsStateandLastSeenfields are written under the peer's lock, so callers reading them raced with those writes.Fix: both return copies taken under the lock. The type and signatures are unchanged.
6. Found while testing:
GenerateFrameIDrepeats under concurrency. It returned only a nanosecond timestamp. Clock resolution is coarser than that on some platforms (microseconds on macOS), so concurrent requests got the same correlation ID and one lost its response. IDs now end with a sequence number shared across the process. The client package and the server's connection IDs use this function too, so they also benefit.Also:
AddPeerwith an ID that already exists now disconnects the peer it replaces, and a reconnect stops once its peer has been removed or replaced.Tests
New tests in
dwp/federation_lifecycle_test.goanddwp/frame_test.go:TestFederation_StopWithConnectedPeerTestFederation_StopDuringReconnectBackoffTestFederation_StopDuringReconnectHandshake: the peer accepts the socket and never answers authTestFederation_DoubleStartTestFederation_StopIsFinalTestFederation_TimeoutReconnectsOnce: checks that the peer accepted exactly 2 connections and has exactly 1 openTestFederation_ConcurrentWritesDoNotInterleave: 32 goroutines × 25 forwards, plus 1ms heartbeats; the fake peer checks every frameTestFederation_PeerSnapshotsDoNotRaceTestGenerateFrameIDConcurrentUniqueAgainst the old
federation.go/frame.go, every new test fails. The Stop tests hang. The timeout test sees 7 connections accepted and 5 open. The write test sees malformed frames and broken pipes. The frame-ID test finds a duplicate.TestFederation_StopIsFinalcan't compile against the oldStartsignature, so it was left out of that run.With the fix:
go vet ./dwp/passes,golangci-lint run ./dwp/...reports 0 issues,go test -race -count=1 ./dwp/...passes, and the new tests pass at-count=10with-race.