Repository navigation
fix(engine): make Start idempotent and retryable after a failure - #55
Merged
Merged
Conversation
Engine.Start had no guard against a second call once it had succeeded. A repeat Start ran wfRunner.ResumeAll again, re-executing workflow runs in "running" state that this process was still executing, and called StartWakeListener again, overwriting wakeStop and leaking the first listener (a Postgres LISTEN connection and its goroutine). The forge extension guards with IsStarted, but direct engine users did not. Start now records success under the lifecycle gate and a later Start returns nil without doing anything. Start after Stop or StopWorkers still returns durable.ErrClosed. A Start that failed after launching the durable worker cleaned up with stopDurable, which marks durable workflows permanently stopped, so every later Start returned durable.ErrClosed. The worker itself cannot be reused either: drt.Worker runs once and its Stop is final. Start now checks up front that the durable worker can launch and only launches it after every fallible step (resume, cron scheduler, dispatcher pool) has succeeded. A failed Start therefore never touches the durable worker, and its cleanup only stops the cron scheduler, so a retry starts everything normally. Stop's terminal semantics are unchanged.
Codecov Report❌ Patch coverage is
📢 Thoughts on this report? Let us know! |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Bug
Repeat Start.
Engine.Startonly checkedstopping, so a second call after a successful Start ran the whole boot again:wfRunner.ResumeAllre-executed workflow runs inrunningstate that this process was still executing, so their steps ran twice.StartWakeListenerwas subscribed again andwakeStopwas overwritten. The first listener (a Postgres LISTEN connection and its goroutine) leaked.The forge extension guards with
IsStarted, but code that calls the engine directly had no such guard.Failed Start. With durable workflows enabled,
startDurablelaunched the worker first. If a later step failed (the cron scheduler or the dispatcher pool), the deferred cleanup calledstopDurable, which setsstopped = true. After that every Start returneddurable.ErrClosed, so the engine could not be started again. The worker object can't be reused either:drt.Worker.RunreturnsErrWorkerStartedon a second call, and itsStopis final.Fix
started, under the lifecycle gate. After a successful Start, another Start returns nil and does nothing. Start after Stop or StopWorkers still returnsdurable.ErrClosed.startDurableis split in two:checkDurableStartruns before anything else and can fail (durable already stopped, or ctx cancelled).launchDurablecannot fail. It runs only after resume, the scheduler and the pool have all started.RuntimeID, and the extension's operator runtime holds the*drt.Workerpointer.Tests
New file
engine/start_idempotent_test.go:TestStartTwiceIsNoOp: a store wrapper counts the resume scans (ListRunswithState: running) and theStartWakeListenercalls. The test checks that each happens once across two Starts, that Stop stops one listener, and that Start after Stop returnsErrClosed.TestFailedStartCanBeRetriedWithDurableWorkflows: a pool set withSetPoolfails its first Start. The test checks that the durable worker is stillnot_startedafter the failed Start, that the retried Start succeeds, that the worker reachesWorkerRunning, and that Health passes.Both tests fail on
main:durable: execution closed.go vet ./engine/ ./durable/...andgo test -race -count=1 ./engine/ ./durable/... ./cron/... ./extension/...pass.