Skip to content

fix(engine): make Start idempotent and retryable after a failure - #55

Merged
juicycleff merged 1 commit into
mainfrom
fix/engine-start-idempotent-retry
Nov 19, 2026
Merged

juicycleff merged 1 commit into
mainfrom
fix/engine-start-idempotent-retry

Conversation

@juicycleff

Copy link
Copy Markdown
Contributor

Bug

Repeat Start. Engine.Start only checked stopping, so a second call after a successful Start ran the whole boot again:

  • wfRunner.ResumeAll re-executed workflow runs in running state that this process was still executing, so their steps ran twice.
  • StartWakeListener was subscribed again and wakeStop was overwritten. The first listener (a Postgres LISTEN connection and its goroutine) leaked.

The forge extension guards with IsStarted, but code that calls the engine directly had no such guard.

Failed Start. With durable workflows enabled, startDurable launched the worker first. If a later step failed (the cron scheduler or the dispatcher pool), the deferred cleanup called stopDurable, which sets stopped = true. After that every Start returned durable.ErrClosed, so the engine could not be started again. The worker object can't be reused either: drt.Worker.Run returns ErrWorkerStarted on a second call, and its Stop is final.

Fix

  • Start now records success in started, under the lifecycle gate. After a successful Start, another Start returns nil and does nothing. Start after Stop or StopWorkers still returns durable.ErrClosed.
  • startDurable is split in two:
    • checkDurableStart runs before anything else and can fail (durable already stopped, or ctx cancelled).
    • launchDurable cannot fail. It runs only after resume, the scheduler and the pool have all started.
  • Because of that order, a failed Start never launches the durable worker. Its cleanup only stops the cron scheduler, and a retried Start launches the worker normally.
  • The worker is never rebuilt. Rebuilding it would break the rule that a replacement uses a new RuntimeID, and the extension's operator runtime holds the *drt.Worker pointer.
  • Stop's terminal behaviour is unchanged.

Tests

New file engine/start_idempotent_test.go:

  • TestStartTwiceIsNoOp: a store wrapper counts the resume scans (ListRuns with State: running) and the StartWakeListener calls. The test checks that each happens once across two Starts, that Stop stops one listener, and that Start after Stop returns ErrClosed.
  • TestFailedStartCanBeRetriedWithDurableWorkflows: a pool set with SetPool fails its first Start. The test checks that the durable worker is still not_started after the failed Start, that the retried Start succeeds, that the worker reaches WorkerRunning, and that Health passes.

Both tests fail on main:

  • Second Start: scans 2, listeners 2.
  • Retried Start: durable: execution closed.

go vet ./engine/ ./durable/... and go test -race -count=1 ./engine/ ./durable/... ./cron/... ./extension/... pass.

Engine.Start had no guard against a second call once it had succeeded.
A repeat Start ran wfRunner.ResumeAll again, re-executing workflow runs
in "running" state that this process was still executing, and called
StartWakeListener again, overwriting wakeStop and leaking the first
listener (a Postgres LISTEN connection and its goroutine). The forge
extension guards with IsStarted, but direct engine users did not.

Start now records success under the lifecycle gate and a later Start
returns nil without doing anything. Start after Stop or StopWorkers
still returns durable.ErrClosed.

A Start that failed after launching the durable worker cleaned up with
stopDurable, which marks durable workflows permanently stopped, so every
later Start returned durable.ErrClosed. The worker itself cannot be
reused either: drt.Worker runs once and its Stop is final.

Start now checks up front that the durable worker can launch and only
launches it after every fallible step (resume, cron scheduler,
dispatcher pool) has succeeded. A failed Start therefore never touches
the durable worker, and its cleanup only stops the cron scheduler, so a
retry starts everything normally. Stop's terminal semantics are
unchanged.
@codecov

codecov Bot commented Nov 19, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 93.33333% with 1 line in your changes missing coverage. Please review.

Files with missing lines Patch % Lines
engine/durable.go 87.50% 1 Missing ⚠️

📢 Thoughts on this report? Let us know!

@juicycleff
juicycleff merged commit 3f4ace4 into main Nov 19, 2026
18 checks passed
@juicycleff
juicycleff deleted the fix/engine-start-idempotent-retry branch November 19, 2026 05:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant