Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,14 @@ Show the list to the human and ask them to classify each file. Pass
`--allow-new-files` only once they confirm every file is genuinely new. For a
rename, see [Renaming](#naming-and-renaming); delete stale files.

**When a deploy stops at the push preflight** ("Push preflight failed …
Nothing was written to the remote API"), a resource the push would write (one
you selected, or a dependency it would create) has a validation error or an
unresolved credential. Fix the listed findings and rerun; nothing was applied,
so there is nothing to roll back. Errors in files outside the push only warn
(`--strict` blocks those too). The preflight only catches what `validate`
knows about, so an API rejection can still leave a push partly applied.

If something goes wrong: `npm run rollback -- <org> --list`, then (with a yes)
`--to <timestamp>`. Details for every workflow:
[Everyday workflows](docs/guides/workflows.md);
Expand Down
4 changes: 2 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -207,9 +207,9 @@ More detail: [How the engine works](docs/guides/how-it-works.md).
| --- | --- |
| `npm run setup` | Connect an org: create `.env.<org>` and `resources/<org>/` |
| `npm run validate -- <org>` | Check resource files offline before deploying |
| `npm run apply -- <org>` | Deploy: pull, merge, then push (the default) |
| `npm run apply -- <org>` | Deploy: pull, merge, then preflight and push (the default) |
| `npm run pull -- <org>` | Sync platform changes down without overwriting local edits |
| `npm run push -- <org>` | Push without pulling first (prefer `apply`) |
| `npm run push -- <org>` | Preflight the write plan, then push without pulling first (prefer `apply`); `--strict` also blocks validation errors outside the plan |
| `npm run rollback -- <org>` | Restore a pre-deploy snapshot |
| `npm run cleanup -- <org>` | Find, and optionally delete, platform resources with no file |
| `npm run audit -- <org>` | Report drift between files, state and the platform |
Expand Down
4 changes: 2 additions & 2 deletions docs/guides/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -11,9 +11,9 @@ The other commands are direct only.
| --- | --- | --- |
| `npm run setup` | `npm run setup [-- <org>]` | Connect an org: creates `.env.<org>` and `resources/<org>/`. |
| `npm run validate` | `npm run validate -- <org>` | Check resource files offline: API shape rules, and that every reference names a file or a state entry. `apply` runs it first. On GitHub Actions, findings are also shown on the files in the pull request. |
| `npm run apply` | `npm run apply -- <org> [types or paths]` | **The default deploy:** pull, merge, then push. See [workflows](workflows.md). |
| `npm run apply` | `npm run apply -- <org> [types or paths]` | **The default deploy:** pull, merge, then push. The push preflight below runs before remote writes. See [workflows](workflows.md). |
| `npm run pull` | `npm run pull -- <org> [--force] [--bootstrap]` | Sync platform changes down; never overwrites local edits unless `--force`. |
| `npm run push` | `npm run push -- <org> [--dry-run] [--strict]` | Push without pulling first. Prefer `apply`. `--strict` aborts before any API call if validation finds an error. |
| `npm run push` | `npm run push -- <org> [--dry-run] [--strict]` | Push without pulling first. Prefer `apply`. Preflight blocks validation errors and unresolved credential names in the write plan (selected resources plus auto-applied dependencies) before any remote mutation, including deletions. `--dry-run` uses the same gate. `--strict` additionally blocks error-severity findings outside the plan; warnings remain advisory. Bootstrap reads may run before preflight. |
| `npm run rollback` | `npm run rollback -- <org> --list` or `--to <ISO>` | Restore a pre-deploy snapshot from `.vapi-state.<org>.snapshots/`. |
| `npm run cleanup` | `npm run cleanup -- <org> [--force --confirm <org>]` | List platform resources with no file; delete them only with both flags. |
| `npm run audit` | `npm run audit -- <org> [--type <type>]` | Report drift between files, state and the platform. Exits 1 on any finding, so it can run in CI. |
Expand Down
19 changes: 19 additions & 0 deletions improvements.md
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,25 @@

---

## Push write-plan preflight [RESOLVED 2026-11-20]

- **Problem:** Push is not atomic: a 400 on a later payload can leave earlier
resources applied and the org partially updated.
- **Current behavior:** Verified in `src/push.ts`: validation now gates all
selected writes and local auto-applied dependencies before orphan deletion
or POST/PATCH. `src/push-plan.ts` computes this plan without API calls.
- **Risk:** Previously, known error-severity findings only warned unless
`--strict` was set. Unknown API-side errors can still leave partial writes;
this is preflight validation, not a remote transaction.
- **Current mitigation:** Default preflight (also in dry-run) blocks planned
errors and unresolved credential names. `--strict` also blocks errors in
unrelated local resources. Bootstrap pulls before the gate only read remote
state.
- **Possible fix:** Extend offline validators as new API rejection cases are
discovered; true atomicity would require platform transaction support.
- **Status:** Resolved for known validation findings; covered by local HTTP
stub tests in `tests/push-preflight.test.ts` asserting zero remote mutations.

## How to read this file

Sections are ordered by **severity / blast radius**, not by date discovered.
Expand Down
74 changes: 74 additions & 0 deletions src/push-plan.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,74 @@
import { extractReferencedIds } from "./resolver.ts";
import type {
LoadedResources,
ResourceFile,
ResourceType,
StateFile,
} from "./types.ts";
import { VALID_RESOURCE_TYPES } from "./types.ts";
import type { ValidationFinding } from "./validate.ts";

const UUID_RE =
/^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i;

// Mirror ensure*Exists without fetching inventories or adopting remote IDs.
export function computePushPlan(
selected: LoadedResources,
local: LoadedResources,
state: StateFile,
): Set<string> {
const plan = new Set<string>();
const available = new Map<ResourceType, Set<string>>();
for (const type of VALID_RESOURCE_TYPES) {
available.set(type, new Set(Object.keys(state[type])));
for (const resource of selected[type]) {
plan.add(`${type}:${resource.resourceId}`);
available.get(type)!.add(resource.resourceId);
}
}
const ensure = (type: "tools" | "structuredOutputs", id: string): boolean => {
if (UUID_RE.test(id) || available.get(type)!.has(id)) return false;
if (!local[type].some((r) => r.resourceId === id)) return false;
plan.add(`${type}:${id}`);
available.get(type)!.add(id);
return true;
};
const assistantDeps = (assistant: ResourceFile<Record<string, unknown>>) => {
const refs = extractReferencedIds(assistant.data);
let created = false;
for (const id of refs.tools) created = ensure("tools", id) || created;
for (const id of refs.structuredOutputs)
created = ensure("structuredOutputs", id) || created;
return created;
};
for (const assistant of selected.assistants) assistantDeps(assistant);
for (const squad of selected.squads) {
for (const id of extractReferencedIds(squad.data).assistants) {
if (UUID_RE.test(id)) continue;
const assistant = local.assistants.find((r) => r.resourceId === id);
if (!assistant) continue;
const depsCreated = assistantDeps(assistant);
if (depsCreated || !available.get("assistants")!.has(id)) {
plan.add(`assistants:${id}`);
available.get("assistants")!.add(id);
}
}
}
return plan;
}

export function pushBlockingFindings(
findings: ValidationFinding[],
plan: Set<string>,
strict: boolean,
): ValidationFinding[] {
return findings
.filter(
(finding) =>
(finding.severity === "error" && strict) ||
(plan.has(`${finding.type}:${finding.resourceId}`) &&
(finding.severity === "error" ||
finding.rule === "unresolved-credential")),
)
.map((finding) => ({ ...finding, severity: "error" }));
}
58 changes: 42 additions & 16 deletions src/push.ts
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ import { checkDriftForUpdate, type DriftCheckResult } from "./drift.ts";
import { deleteBaseline, writeBaseline } from "./hash-store.ts";
import { assertStateMigrated } from "./migrate-hash-store.ts";
import { detectOrphanYamls, formatGateMessage } from "./new-file-gate.ts";
import { computePushPlan, pushBlockingFindings } from "./push-plan.ts";
import {
formatRecanonicalizeReport,
recanonicalizeStateKeys,
Expand Down Expand Up @@ -1767,10 +1768,7 @@ async function main(): Promise<void> {
}
}

// Run client-side validators against the loaded resource set. In default
// mode, errors are surfaced as warnings so a single bad spec doesn't block
// an otherwise-good push. With --strict, any error-severity finding aborts
// before any API call.
// Validate the full local set; gate errors against the write plan below.
console.log("\n🔎 Running validators...");
const findings = [
...validateResources(loadedResources),
Expand All @@ -1791,18 +1789,6 @@ async function main(): Promise<void> {
} else {
console.log(" ✅ No validation issues.");
}
const errorCount = findings.filter((f) => f.severity === "error").length;
if (errorCount > 0) {
if (STRICT_VALIDATION) {
console.error(
`\n❌ Validation failed (${errorCount} error(s)). --strict refuses to push. Fix the issues above or drop --strict.`,
);
process.exit(1);
}
console.warn(
` ⚠️ ${errorCount} validation error(s) detected — push will continue (use --strict to abort on errors).`,
);
}

// Resolve credential names → UUIDs in all resource data before applying
const credMap = credentialForwardMap(state);
Expand Down Expand Up @@ -1890,6 +1876,46 @@ async function main(): Promise<void> {
? filterResourcesByPaths(allEvals, "evals")
: [];

const plan = computePushPlan(
{
tools,
structuredOutputs,
assistants,
squads,
personalities,
scenarios,
simulations,
simulationSuites,
evals,
},
{
tools: allTools,
structuredOutputs: allStructuredOutputs,
assistants: allAssistants,
squads: allSquads,
personalities: allPersonalities,
scenarios: allScenarios,
simulations: allSimulations,
simulationSuites: allSimulationSuites,
evals: allEvals,
},
state,
);
const blocking = pushBlockingFindings(findings, plan, STRICT_VALIDATION);
if (blocking.length > 0) {
console.error(summarizeFindings(blocking));
console.error(
`\n❌ Push preflight failed (${blocking.length} blocking finding(s))${STRICT_VALIDATION ? " (--strict)" : ""}. Nothing was written to the remote API.`,
);
process.exit(1);
}
const outsideErrors = findings.filter((f) => f.severity === "error").length;
if (outsideErrors > 0) {
console.warn(
` ⚠️ ${outsideErrors} validation error(s) outside the push plan — push will continue (use --strict to abort on all errors).`,
);
}

// Scoped run: print exactly what's in scope (the load above was quiet).
if (partial) {
const scoped: Array<[string, ResourceFile<Record<string, unknown>>[]]> = [
Expand Down
7 changes: 4 additions & 3 deletions src/validate.ts
Original file line number Diff line number Diff line change
Expand Up @@ -2,9 +2,10 @@
// Client-side validators — fail-fast schema / lockstep / shape checks
//
// Designed to catch the classes of errors that would otherwise only surface
// when the Vapi API returns a 400 mid-push. The push pipeline runs these in
// warning-only mode by default; `--strict` promotes warnings to blocking
// errors that abort before any API call.
// when the Vapi API returns a 400 mid-push. Push blocks error-severity findings
// in its write plan (including auto-applied dependencies) before any remote
// mutation, also in dry-run. `--strict` additionally blocks errors outside
// the plan; warning-severity findings remain advisory.
//
// Sources for each check are documented in `improvements.md`:
// - Name length cap (40 chars) → improvements #18
Expand Down
Loading
Loading