Skip to content

fix(dwp): make federation lifecycle, reconnects and writes concurrency-safe - #57

Merged
juicycleff merged 1 commit into
mainfrom
fix/dwp-federation-lifecycle
Nov 19, 2026
Merged

juicycleff merged 1 commit into
mainfrom
fix/dwp-federation-lifecycle

Conversation

@juicycleff

Copy link
Copy Markdown
Contributor

Summary

dwp.Federation had races that made Stop hang and could corrupt the stream to a peer. This PR fixes them and adds tests that drive a fake WebSocket peer.

Bugs, triggers and fixes

1. Double Start leaks loops. Each Start overwrote f.cancel, so a second call left the first heartbeat and health loops running with nothing to cancel them. Stop then waited on them forever.
Fix: Start is a no-op while running. Stop is final: Start and AddPeer return ErrFederationStopped after it. Start now returns error; existing call sites still compile.

2. Stop hangs with any connected or reconnecting peer. Stop called wg.Wait() before it closed peer connections. The WaitGroup held peer read loops blocked in ReadServerText and reconnects stuck in time.Sleep or a dial with context.Background.
Fix: Stop cancels a context the federation owns, closes every peer connection, and only then waits. Reconnects wait on a timer that also watches that context. connectPeer ties the dial and the auth exchange to the same context (it closes the socket to interrupt the auth read), and the handshake is capped at 30s. The connection is installed and its read loop added under the lifecycle lock. So nothing joins the WaitGroup once Stop has started, and no connection is installed after Stop or RemovePeer has retired the peer.

3. A heartbeat timeout opens two connections. checkPeerHealth closed the connection and started a reconnect. The read loop of that closed connection started a second one. Both dialled, which leaked a connection and ran two read loops on one conn.
Fix: a per-peer reconnecting flag allows one reconnect at a time. The health check closes the connection inside the same critical section as its timeout check. Each read loop only acts on the connection it was started with, so an old loop exits quietly and doesn't change the new connection's state.

4. Concurrent writes interleave frames. writeJSONFrame writes a frame header and its payload as two separate writes with no lock. Heartbeats and ForwardJob/ForwardEvent/Broadcast* write at the same time, so their bytes interleaved and the peer dropped the connection.
Fix: all writes to a peer go through a per-peer write mutex.

5. GetPeer/Peers returned the live *Peer. Its State and LastSeen fields are written under the peer's lock, so callers reading them raced with those writes.
Fix: both return copies taken under the lock. The type and signatures are unchanged.

6. Found while testing: GenerateFrameID repeats under concurrency. It returned only a nanosecond timestamp. Clock resolution is coarser than that on some platforms (microseconds on macOS), so concurrent requests got the same correlation ID and one lost its response. IDs now end with a sequence number shared across the process. The client package and the server's connection IDs use this function too, so they also benefit.

Also: AddPeer with an ID that already exists now disconnects the peer it replaces, and a reconnect stops once its peer has been removed or replaced.

Tests

New tests in dwp/federation_lifecycle_test.go and dwp/frame_test.go:

  • TestFederation_StopWithConnectedPeer
  • TestFederation_StopDuringReconnectBackoff
  • TestFederation_StopDuringReconnectHandshake: the peer accepts the socket and never answers auth
  • TestFederation_DoubleStart
  • TestFederation_StopIsFinal
  • TestFederation_TimeoutReconnectsOnce: checks that the peer accepted exactly 2 connections and has exactly 1 open
  • TestFederation_ConcurrentWritesDoNotInterleave: 32 goroutines × 25 forwards, plus 1ms heartbeats; the fake peer checks every frame
  • TestFederation_PeerSnapshotsDoNotRace
  • TestGenerateFrameIDConcurrentUnique

Against the old federation.go/frame.go, every new test fails. The Stop tests hang. The timeout test sees 7 connections accepted and 5 open. The write test sees malformed frames and broken pipes. The frame-ID test finds a duplicate. TestFederation_StopIsFinal can't compile against the old Start signature, so it was left out of that run.

With the fix: go vet ./dwp/ passes, golangci-lint run ./dwp/... reports 0 issues, go test -race -count=1 ./dwp/... passes, and the new tests pass at -count=10 with -race.

…y-safe

Federation had several races that made Stop hang and could corrupt the
peer stream.

- Start overwrote f.cancel unconditionally, so a second Start left the
  first heartbeat and health loops running with no way to cancel them,
  and Stop then waited on them forever. Start is now a no-op while
  running. Stop is final: Start and AddPeer return ErrFederationStopped
  afterwards.

- Stop waited on the WaitGroup before closing peer connections, but the
  group held peer read loops blocked in ReadServerText and reconnects
  sleeping on time.Sleep with context.Background. Stop now cancels a
  federation-owned context, closes every peer connection, and only then
  waits. Reconnects wait on a timer selected against that context, and
  connectPeer ties the dial and the auth exchange to it (closing the
  socket to interrupt the auth read), with a 30s handshake bound.
  Installing a connection and adding its read loop happen under the
  lifecycle lock, so nothing is added to the WaitGroup once Stop has
  begun and a connection cannot be installed after Stop or RemovePeer
  has retired the peer.

- On a heartbeat timeout, checkPeerHealth started a reconnect and the
  read loop of the connection it closed started a second one; both
  dialled, leaking a connection and running two read loops on one conn.
  A per-peer reconnecting flag now allows a single reconnect at a time,
  the health check tears the connection down in the same critical
  section as its check, and each read loop owns only the conn it was
  started with, so a stale loop exits quietly instead of clobbering the
  new connection's state.

- writeJSONFrame writes a WebSocket frame as a header and a payload with
  no lock, and heartbeats and forwarded requests write concurrently, so
  frames interleaved and the peer dropped the connection. Writes now go
  through a per-peer write mutex.

- GenerateFrameID was a bare nanosecond timestamp. Clock resolution is
  coarser than that on some platforms, so concurrent requests shared an
  ID and one of them lost its response. IDs now carry a process-wide
  sequence number.

- GetPeer and Peers handed out the live *Peer whose State and LastSeen
  are written under its lock. They now return copies taken under the
  lock; the type and signatures are unchanged.

AddPeer with an existing ID now disconnects the peer it replaces, and a
reconnect stops once its peer is removed or replaced.

Tests use a fake WebSocket peer and cover Stop with a connected peer, in
reconnect backoff and in a stalled reconnect handshake; double Start;
Stop being final; a timeout producing exactly one reconnect; concurrent
forwards arriving as well-formed frames; race-free peer snapshots; and
unique frame IDs under concurrency.
@juicycleff
juicycleff merged commit 29c3277 into main Nov 19, 2026
17 checks passed
@juicycleff
juicycleff deleted the fix/dwp-federation-lifecycle branch November 19, 2026 05:16
@codecov

codecov Bot commented Nov 19, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 86.33094% with 19 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
dwp/federation.go 86.13% 19 Missing ⚠️

📢 Thoughts on this report? Let us know!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant