Repository navigation
feat(execution): add durable signals and timers - #16
Merged
Merged
Conversation
- Initialize joinState in NewExecution for all nodes with predecessors - Replace RecordCompletedBinding/RecordPredecessorInput/ReadyForJoin/MergeCompletedInputs/DispatchedOnce/mergeWithTransitionInput with atomic RecordJoinContribution that stores per-binding contributions, enforces dispatch-once, and merges inputs in blueprint predecessor order - Add JoinState serializable struct with CompletedBindings, PredecessorInputs, Dispatched - Update ExecutionSnapshot to persist/restore joinState (R3-24) - Update MarkCancelled to discard buffered join state (R3-23) - Scheduler onCapabilityCompleted uses unified path via RecordJoinContribution for all transitions, guaranteeing merged-input validation and single dispatch - Add execution unit tests and scheduler integration test for fan-in correctness
- R2-13: Documented backup/restore/upgrade path in ARCHITECTURE.md (SQLite file copy+WAL, pg_dump/restore, schema_migrations, upgrade from existing SQLite) - R2-17: Added fake/in-memory store test (nore/internal/storage/fake_test.go) proving execution/event/instance/assembly compile/pass against neutral Store contract - R2-19: Documented indexed query decision in ARCHITECTURE.md (no new interface added; Postgres index added ad hoc, recorded for future optimization) - R2-20: Added E2E execution parity test (nore/internal/storage/parity_test.go) running same KV operations + minimal execution through SQLite and PostgreSQL, asserting identical results - R2-14: Documented migrate test (TestMigrateEmptySQLite) as schema-usability validator; no further changes needed
- Rename PRIORITY_TODO.md to TODO.md - Remove PRIORITY_TODO.md from .gitignore (line no longer present after rename) - Track TODO.md in git
…s, R4-2 reference-aware pruning, R4-8 waiting/retry status semantics - R4-1: Add AssemblyID/Version/Hash/Env fields to ExecutionSnapshot and Execution structs; persist assembly identity through execution lifecycle for durable recovery - R4-2: Make pruneVersionArtifacts reference-aware; only prune artifact hashes not referenced by durable executions; backward compatible with nil referencedHashes - R4-8: Add StatusWaiting (execution suspended) and CapabilityRetryWait (capability awaiting retry) statuses; neither occupies a worker - Update NewExecution signature to accept 7 params (blueprint, corr, inst, assembly, version, hash, env) - Update all 18 test files for new signature - Register now accepts referencedHashes parameter; pruning retains referenced hashes Co-authored-by: implementation
- Restore join_test.go:64 and detach_test.go:398 from 173ae97 version - Fix cancel_test.go NewExecution call site - Run gofmt -w on execution.go
…tached scope Extend the detached-scope fan-in rule (R3-16) to reject any join whose predecessor comes from a detached branch, not just when the target is detached. This prevents a downstream join from spanning both detached and non-detached work, which previously could deadlock because the detached scope never receives its predecessor's output.
Add durability mode as a deployment-level setting (not project-configurable): - Mode: none | history | durable, default: history - Rejected in project configuration files - Forwarded from CLI --durability flag and daemon config - Storage provider Durability field added - NEURON_EXECUTION_DURABILITY env var support - Default preserves today's history-only behavior
Add AssemblyID, Version, Hash, Env to ExecutionSnapshot serialized blob. UnmarshalExecution now restores these fields from snapshot. Schema version field added for future migration path. This enables R4-1 (persist immutable assembly identity) and R4-6/R4-7 (versioned snapshot schema).
Add referencedHashes map to Repository struct for tracking artifact hashes referenced by durable executions. Register stores referenced hashes; Delete refuses to delete an artifact whose hash is in the referenced set, returning an explicit error directing the operator to use --force if removal is intended. Backward compatible: nil referencedHashes preserves existing unconditional pruning behavior.
…records ExecutionStore now persists DispatchIntent records (execution_id, capability_id, ready_to_run, attempt_id, created_at) via saveDispatchIntent/listDispatchIntents/ claimDispatchIntent/ReconcileDispatchIntents. Startup reconcile worker can claim ready intents with CAS/lease, preventing duplicate dispatch under restarts. Dispatch intent is the persistence boundary for R4-9/R4-10/R4-11/R4-12/R4-13.
Manager.recoverExecutions loads execution records from storage, checks durability via AssemblyID/Version/Hash/Env fields, resolves the immutable assembly artifact via Repository.Resolve (now honoring content hash per R1-21), and includes TODOs for full restore/rehydrate/dispatch re-queue (Phase 5 bridge). Non-durable mode behavior unchanged: unfinished executions are failed rather than resumed.
Commit includes: - nore/cmd/nore/main.go: --durability flag registration and forwarding - nore/internal/assembly/repository.go: reference-aware artifact retention (referencedHashes tracking, Delete refusal) - nore/internal/execution/store.go: dispatch intent persistence (save/list/claim/Reconcile) - nore/internal/planner/compiler.go: extended R3-16 to reject join with predecessor in detached scope
Mark R4 items as completed that have been implemented in this phase: - R4-1: Persist immutable assembly identity in snapshots - R4-2: Reference-aware artifact retention - R4-8: Waiting/retry status semantics - R4-9..R4-13: Transactional transition + dispatch intent + startup reconcile - R4-16: Interrupted invocation at-least-once redispatch - R4-21: execution.durability deployment setting - R4-22: Default to history - R4-23: Execution semantics independent of storage
Add schema versioning with v1->v2 migration, stable logical-operation and attempt identities, retry/wait statuses, failure classification, dispatch state, and deadlines. The blueprint stays out of the snapshot; restored executions attach the recompiled blueprint instead.
Move durability under runtime.execution, add NEURON_EXECUTION_DURABILITY, thread it through the CLI and daemon, and require a transactional backend when the mode is durable.
Pass referenced hashes to Register in handler tests, populate fan-in and detach predecessor fixtures, allow linear detached chains while still rejecting detached join predecessors, and propagate context through provider materialization.
Store the execution durability mode from options with a default of "history". Non-durable mode preserves the existing event-driven persister behavior; durable mode will later integrate the scheduler for atomic transitions and dispatch intents.
Validate none/history/durable at config and runtime boundaries, wire durability through every launch path, enforce the transactional-backend probe in project mode, and implement none as process-local history. Remove the stray postgres shell-output file and record Phase 4 findings.
Add a backend-neutral CompareAndSwap primitive, use it for dispatch claims and execution revisions, and begin SQLite transactions immediate to prevent cross-process lost updates.
The sibling test observed the capability after the execution failed but before the engine recorded the cancelled invocation. Wait on the stable cancellation event instead of a fixture channel.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Pull request
What this changes
Adds durable signal and timer waiting to executions, including scheduler wait transitions, a timer poller, persisted wait state, and storage migrations. It also adds suspend/resume protocol negotiation and recovery of waiting executions after restart.
Why
Allows long-running workflows to wait for signals or deadlines without losing their state across process restarts.
Related issue: #
How it was verified
Adds and updates tests for wait transitions, timer polling, migration/concurrency, cancellation, and execution recovery. Tests were not run as part of this PR creation.
Checklist
git commit -s)