Skip to content

fix(worker): give each pool run fresh lifecycle state - #56

Merged
juicycleff merged 1 commit into
mainfrom
fix/worker-pool-restart
Nov 19, 2026
Merged

juicycleff merged 1 commit into
mainfrom
fix/worker-pool-restart

Conversation

@juicycleff

Copy link
Copy Markdown
Contributor

Bug

worker.Pool created stopCh once in NewPool and closed it in Stop. Nothing recreated it in Start.

Trigger

Start, Stop, then Start again on the same pool. Every loop in the second run (fetcher, workers, heartbeat, reaper) selects on the already-closed channel and returns at once. The pool reports itself running but processes nothing. The next Stop then panics with close of closed channel.

Stop also cleared running and released the lock before waiting for the old run's goroutines. That let a concurrent Start launch a new run whose fields raced the goroutines still draining, which -race reports.

Fix

Same approach as the cron.Scheduler fix:

  • A new lifecycleMu serializes Start and Stop.
  • Start is a no-op while the pool is running. Otherwise it builds fresh per-run state: the stop channel, the cancel context, and the job hand-off and slot channels. It also resets the readiness poll result so Ready describes the new run, and drops any leftover Wake nudge, which the new fetcher's immediate first poll already covers.
  • Stop is a no-op when the pool isn't running. Otherwise it closes the channel, cancels the context and waits for every goroutine of the run while still holding lifecycleMu. It then throws away any in-flight record left behind (normally there are none) so a lease timer can't outlive its run.
  • Calling Stop twice, which Engine and Dispatcher both do on shutdown, is still a safe no-op. A concurrent second Stop now waits for the first to finish instead of returning while the pool is still draining.

Tests

New tests in worker/pool_restart_test.go. The pool under test has the heartbeat, reaper and reclaim sweep enabled.

  • TestPool_RestartProcessesNewJob: start, run a job, stop, start again, and a newly enqueued job completes. Ready is an error while stopped and nil after the restart.
  • TestPool_StopTwiceAfterRestart: three start/stop cycles, each with a second Stop, and none of them panic.
  • TestPool_ConcurrentStartStop: 8 goroutines interleave Start/Stop, then the pool still processes a job.

Results on main without the fix: the first test times out (the second job never runs) and then panics on Stop with close of closed channel. The second panics with close of closed channel. The third reports data races and then hangs.

With the fix, these pass with -race:

  • go vet ./worker/
  • go test -race -count=1 ./worker/... ./engine/ .
  • the pool tests run 10 times with -count=10

Pool created its stop channel once in NewPool and closed it in Stop.
After Stop and a later Start, every loop of the new run selected on the
already-closed channel and returned at once, so the pool reported itself
running while processing nothing, and the next Stop panicked with
"close of closed channel". Stop also released the lock before waiting
for the old run, so a concurrent Start could launch a new run whose
fields raced the draining goroutines.

Follow the cron.Scheduler fix: a lifecycleMu serializes Start and Stop.
Start is a no-op while running and otherwise builds fresh per-run state
(stop channel, cancel context, job hand-off and slot channels), resets
the readiness poll result, and drops any stale Wake nudge. Stop is a
no-op when not running, otherwise closes and cancels, waits for every
goroutine of the run while still holding lifecycleMu, and then discards
any in-flight record left behind so no lease timer outlives its run.
Calling Stop more than once, as Engine and Dispatcher both do, stays a
no-op.

Tests cover a restart processing a newly enqueued job, Stop twice after
each of several restarts, and concurrent Start/Stop calls under -race.
@codecov

codecov Bot commented Nov 19, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 72.00000% with 7 lines in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
worker/pool.go 72.00% 7 Missing ⚠️

📢 Thoughts on this report? Let us know!

@juicycleff
juicycleff merged commit 723942f into main Nov 19, 2026
18 checks passed
@juicycleff
juicycleff deleted the fix/worker-pool-restart branch November 19, 2026 05:15
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant