Repository navigation
fix(worker): give each pool run fresh lifecycle state - #56
Merged
Merged
Conversation
Pool created its stop channel once in NewPool and closed it in Stop. After Stop and a later Start, every loop of the new run selected on the already-closed channel and returned at once, so the pool reported itself running while processing nothing, and the next Stop panicked with "close of closed channel". Stop also released the lock before waiting for the old run, so a concurrent Start could launch a new run whose fields raced the draining goroutines. Follow the cron.Scheduler fix: a lifecycleMu serializes Start and Stop. Start is a no-op while running and otherwise builds fresh per-run state (stop channel, cancel context, job hand-off and slot channels), resets the readiness poll result, and drops any stale Wake nudge. Stop is a no-op when not running, otherwise closes and cancels, waits for every goroutine of the run while still holding lifecycleMu, and then discards any in-flight record left behind so no lease timer outlives its run. Calling Stop more than once, as Engine and Dispatcher both do, stays a no-op. Tests cover a restart processing a newly enqueued job, Stop twice after each of several restarts, and concurrent Start/Stop calls under -race.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
worker.PoolcreatedstopChonce inNewPooland closed it inStop. Nothing recreated it inStart.Trigger
Start,Stop, thenStartagain on the same pool. Every loop in the second run (fetcher, workers, heartbeat, reaper) selects on the already-closed channel and returns at once. The pool reports itself running but processes nothing. The nextStopthen panics withclose of closed channel.Stopalso clearedrunningand released the lock before waiting for the old run's goroutines. That let a concurrentStartlaunch a new run whose fields raced the goroutines still draining, which-racereports.Fix
Same approach as the
cron.Schedulerfix:lifecycleMuserializesStartandStop.Startis a no-op while the pool is running. Otherwise it builds fresh per-run state: the stop channel, the cancel context, and the job hand-off and slot channels. It also resets the readiness poll result soReadydescribes the new run, and drops any leftoverWakenudge, which the new fetcher's immediate first poll already covers.Stopis a no-op when the pool isn't running. Otherwise it closes the channel, cancels the context and waits for every goroutine of the run while still holdinglifecycleMu. It then throws away any in-flight record left behind (normally there are none) so a lease timer can't outlive its run.Stoptwice, whichEngineandDispatcherboth do on shutdown, is still a safe no-op. A concurrent secondStopnow waits for the first to finish instead of returning while the pool is still draining.Tests
New tests in
worker/pool_restart_test.go. The pool under test has the heartbeat, reaper and reclaim sweep enabled.TestPool_RestartProcessesNewJob: start, run a job, stop, start again, and a newly enqueued job completes.Readyis an error while stopped and nil after the restart.TestPool_StopTwiceAfterRestart: three start/stop cycles, each with a secondStop, and none of them panic.TestPool_ConcurrentStartStop: 8 goroutines interleaveStart/Stop, then the pool still processes a job.Results on
mainwithout the fix: the first test times out (the second job never runs) and then panics onStopwithclose of closed channel. The second panics withclose of closed channel. The third reports data races and then hangs.With the fix, these pass with
-race:go vet ./worker/go test -race -count=1 ./worker/... ./engine/ .-count=10