diff --git a/AGENTS.md b/AGENTS.md index ccdb627..a502508 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -194,6 +194,14 @@ are needed to remove a verified unchanged leftover of an already-adopted resource. The engine never auto-adopts, auto-allows new files or deletes edited files without approval. +**When a deploy stops at the push preflight** ("Push preflight failed … +Nothing was written to the remote API"), a resource the push would write (one +you selected, or a dependency it would create) has a validation error or an +unresolved credential. Fix the listed findings and rerun; nothing was applied, +so there is nothing to roll back. Errors in files outside the push only warn +(`--strict` blocks those too). The preflight only catches what `validate` +knows about, so an API rejection can still leave a push partly applied. + If something goes wrong: `npm run rollback -- --list`, then (with a yes) `--to `. Details for every workflow: [Everyday workflows](docs/guides/workflows.md); diff --git a/README.md b/README.md index e7a7d6f..bd2a1c0 100644 --- a/README.md +++ b/README.md @@ -207,9 +207,9 @@ More detail: [How the engine works](docs/guides/how-it-works.md). | --- | --- | | `npm run setup` | Connect an org: create `.env.` and `resources//` | | `npm run validate -- ` | Check resource files offline before deploying | -| `npm run apply -- ` | Deploy: pull, merge, then push (the default) | +| `npm run apply -- ` | Deploy: pull, merge, then preflight and push (the default) | | `npm run pull -- ` | Sync platform changes down without overwriting local edits; unknown UUIDs get separate files, never name-linked | -| `npm run push -- ` | Push without pulling first (prefer `apply`); review `--dry-run` Name collisions, and use `--adopt [=]` only with explicit approval to take over a resource; unchanged alias files are removed, edited/unbaselined copies need review. Push/apply support `--json` for one grouped link-preflight report (exit 1 on blockers); see [Running in CI](docs/guides/commands.md#running-in-ci). | +| `npm run push -- ` | Preflight the write plan, then push without pulling first (prefer `apply`); `--strict` also blocks validation errors outside the plan. Review `--dry-run` Name collisions, and use `--adopt [=]` only with explicit approval to take over a resource; unchanged alias files are removed, edited/unbaselined copies need review. Push/apply support `--json` for one grouped link-preflight report (exit 1 on blockers); see [Running in CI](docs/guides/commands.md#running-in-ci). | | `npm run rollback -- ` | Restore a pre-deploy snapshot | | `npm run cleanup -- ` | Find, and optionally delete, platform resources with no file | | `npm run audit -- ` | Report drift between files, state and the platform | diff --git a/docs/guides/commands.md b/docs/guides/commands.md index fc64295..70d7e8f 100644 --- a/docs/guides/commands.md +++ b/docs/guides/commands.md @@ -11,9 +11,9 @@ The other commands are direct only. | --- | --- | --- | | `npm run setup` | `npm run setup [-- ]` | Connect an org: creates `.env.` and `resources//`. | | `npm run validate` | `npm run validate -- ` | Check resource files offline: API shape rules, and that every reference names a file or a state entry. `apply` runs it first. On GitHub Actions, findings are also shown on the files in the pull request. | -| `npm run apply` | `npm run apply -- [types or paths] [--adopt [=]]` | **The default deploy:** pull, merge, then push. See [workflows](workflows.md). | +| `npm run apply` | `npm run apply -- [types or paths] [--adopt [=]]` | **The default deploy:** pull, merge, then push. The push preflight below runs before remote writes. See [workflows](workflows.md). | | `npm run pull` | `npm run pull -- [--force] [--bootstrap]` | Sync platform changes down; never overwrites local edits unless `--force`. | -| `npm run push` | `npm run push -- [--dry-run] [--json] [--strict] [--adopt [=]]` | Push without pulling first. Prefer `apply`. `--strict` aborts before any API call if validation finds an error. | +| `npm run push` | `npm run push -- [--dry-run] [--json] [--strict] [--adopt [=]]` | Push without pulling first. Prefer `apply`. Preflight blocks validation errors and unresolved credential names in the write plan (selected resources plus auto-applied dependencies) before any remote mutation, including deletions. `--dry-run` uses the same gate. `--strict` additionally blocks error-severity findings outside the plan; warnings remain advisory. Bootstrap reads may run before preflight. | | `npm run rollback` | `npm run rollback -- --list` or `--to ` | Restore a pre-deploy snapshot from `.vapi-state..snapshots/`. | | `npm run cleanup` | `npm run cleanup -- [--force --confirm ]` | List platform resources with no file; delete them only with both flags. | | `npm run audit` | `npm run audit -- [--type ]` | Report drift between files, state and the platform. Exits 1 on any finding, so it can run in CI. | diff --git a/improvements.md b/improvements.md index 51eac4e..0d98ca7 100644 --- a/improvements.md +++ b/improvements.md @@ -31,6 +31,25 @@ --- +## Push write-plan preflight [RESOLVED 2026-11-20] + +- **Problem:** Push is not atomic: a 400 on a later payload can leave earlier + resources applied and the org partially updated. +- **Current behavior:** Verified in `src/push.ts`: validation now gates all + selected writes and local auto-applied dependencies before orphan deletion + or POST/PATCH. `src/push-plan.ts` computes this plan without API calls. +- **Risk:** Previously, known error-severity findings only warned unless + `--strict` was set. Unknown API-side errors can still leave partial writes; + this is preflight validation, not a remote transaction. +- **Current mitigation:** Default preflight (also in dry-run) blocks planned + errors and unresolved credential names. `--strict` also blocks errors in + unrelated local resources. Bootstrap pulls before the gate only read remote + state. +- **Possible fix:** Extend offline validators as new API rejection cases are + discovered; true atomicity would require platform transaction support. +- **Status:** Resolved for known validation findings; covered by local HTTP + stub tests in `tests/push-preflight.test.ts` asserting zero remote mutations. + ## How to read this file Sections are ordered by **severity / blast radius**, not by date discovered. diff --git a/src/push-plan.ts b/src/push-plan.ts new file mode 100644 index 0000000..83a747e --- /dev/null +++ b/src/push-plan.ts @@ -0,0 +1,74 @@ +import { extractReferencedIds } from "./resolver.ts"; +import type { + LoadedResources, + ResourceFile, + ResourceType, + StateFile, +} from "./types.ts"; +import { VALID_RESOURCE_TYPES } from "./types.ts"; +import type { ValidationFinding } from "./validate.ts"; + +const UUID_RE = + /^[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/i; + +// Mirror ensure*Exists without fetching inventories or adopting remote IDs. +export function computePushPlan( + selected: LoadedResources, + local: LoadedResources, + state: StateFile, +): Set { + const plan = new Set(); + const available = new Map>(); + for (const type of VALID_RESOURCE_TYPES) { + available.set(type, new Set(Object.keys(state[type]))); + for (const resource of selected[type]) { + plan.add(`${type}:${resource.resourceId}`); + available.get(type)!.add(resource.resourceId); + } + } + const ensure = (type: "tools" | "structuredOutputs", id: string): boolean => { + if (UUID_RE.test(id) || available.get(type)!.has(id)) return false; + if (!local[type].some((r) => r.resourceId === id)) return false; + plan.add(`${type}:${id}`); + available.get(type)!.add(id); + return true; + }; + const assistantDeps = (assistant: ResourceFile>) => { + const refs = extractReferencedIds(assistant.data); + let created = false; + for (const id of refs.tools) created = ensure("tools", id) || created; + for (const id of refs.structuredOutputs) + created = ensure("structuredOutputs", id) || created; + return created; + }; + for (const assistant of selected.assistants) assistantDeps(assistant); + for (const squad of selected.squads) { + for (const id of extractReferencedIds(squad.data).assistants) { + if (UUID_RE.test(id)) continue; + const assistant = local.assistants.find((r) => r.resourceId === id); + if (!assistant) continue; + const depsCreated = assistantDeps(assistant); + if (depsCreated || !available.get("assistants")!.has(id)) { + plan.add(`assistants:${id}`); + available.get("assistants")!.add(id); + } + } + } + return plan; +} + +export function pushBlockingFindings( + findings: ValidationFinding[], + plan: Set, + strict: boolean, +): ValidationFinding[] { + return findings + .filter( + (finding) => + (finding.severity === "error" && strict) || + (plan.has(`${finding.type}:${finding.resourceId}`) && + (finding.severity === "error" || + finding.rule === "unresolved-credential")), + ) + .map((finding) => ({ ...finding, severity: "error" })); +} diff --git a/src/push.ts b/src/push.ts index 0067b27..3291b98 100644 --- a/src/push.ts +++ b/src/push.ts @@ -27,6 +27,7 @@ import { deleteBaseline, writeBaseline } from "./hash-store.ts"; import { prepareResourceLinks } from "./link-preflight.ts"; import { assertStateMigrated } from "./migrate-hash-store.ts"; import { NameCollisionReport } from "./name-collisions.ts"; +import { computePushPlan, pushBlockingFindings } from "./push-plan.ts"; import { reconcileStateKeyForResource } from "./reconcile-state-key.ts"; import { writeSnapshot } from "./snapshot.ts"; import { mergeScoped } from "./state-merge.ts"; @@ -1756,10 +1757,7 @@ async function main(): Promise { } } preflightComplete = true; - // Run client-side validators against the loaded resource set. In default - // mode, errors are surfaced as warnings so a single bad spec doesn't block - // an otherwise-good push. With --strict, any error-severity finding aborts - // before any API call. + // Validate the full local set; gate errors against the write plan below. console.log("\nšŸ”Ž Running validators..."); const findings = [ ...validateResources(loadedResources), @@ -1780,18 +1778,6 @@ async function main(): Promise { } else { console.log(" āœ… No validation issues."); } - const errorCount = findings.filter((f) => f.severity === "error").length; - if (errorCount > 0) { - if (STRICT_VALIDATION) { - console.error( - `\nāŒ Validation failed (${errorCount} error(s)). --strict refuses to push. Fix the issues above or drop --strict.`, - ); - process.exit(1); - } - console.warn( - ` āš ļø ${errorCount} validation error(s) detected — push will continue (use --strict to abort on errors).`, - ); - } // Resolve credential names → UUIDs in all resource data before applying const credMap = credentialForwardMap(state); @@ -1879,6 +1865,46 @@ async function main(): Promise { ? filterResourcesByPaths(allEvals, "evals") : []; + const plan = computePushPlan( + { + tools, + structuredOutputs, + assistants, + squads, + personalities, + scenarios, + simulations, + simulationSuites, + evals, + }, + { + tools: allTools, + structuredOutputs: allStructuredOutputs, + assistants: allAssistants, + squads: allSquads, + personalities: allPersonalities, + scenarios: allScenarios, + simulations: allSimulations, + simulationSuites: allSimulationSuites, + evals: allEvals, + }, + state, + ); + const blocking = pushBlockingFindings(findings, plan, STRICT_VALIDATION); + if (blocking.length > 0) { + console.error(summarizeFindings(blocking)); + console.error( + `\nāŒ Push preflight failed (${blocking.length} blocking finding(s))${STRICT_VALIDATION ? " (--strict)" : ""}. Nothing was written to the remote API.`, + ); + process.exit(1); + } + const outsideErrors = findings.filter((f) => f.severity === "error").length; + if (outsideErrors > 0) { + console.warn( + ` āš ļø ${outsideErrors} validation error(s) outside the push plan — push will continue (use --strict to abort on all errors).`, + ); + } + // Scoped run: print exactly what's in scope (the load above was quiet). if (partial) { const scoped: Array<[string, ResourceFile>[]]> = [ diff --git a/src/validate.ts b/src/validate.ts index 10fb3f4..286525c 100644 --- a/src/validate.ts +++ b/src/validate.ts @@ -2,9 +2,10 @@ // Client-side validators — fail-fast schema / lockstep / shape checks // // Designed to catch the classes of errors that would otherwise only surface -// when the Vapi API returns a 400 mid-push. The push pipeline runs these in -// warning-only mode by default; `--strict` promotes warnings to blocking -// errors that abort before any API call. +// when the Vapi API returns a 400 mid-push. Push blocks error-severity findings +// in its write plan (including auto-applied dependencies) before any remote +// mutation, also in dry-run. `--strict` additionally blocks errors outside +// the plan; warning-severity findings remain advisory. // // Sources for each check are documented in `improvements.md`: // - Name length cap (40 chars) → improvements #18 diff --git a/tests/push-preflight.test.ts b/tests/push-preflight.test.ts new file mode 100644 index 0000000..63991f4 --- /dev/null +++ b/tests/push-preflight.test.ts @@ -0,0 +1,243 @@ +import assert from "node:assert/strict"; +import { spawn } from "node:child_process"; +import { + cpSync, + mkdirSync, + mkdtempSync, + rmSync, + symlinkSync, + writeFileSync, +} from "node:fs"; +import { createServer } from "node:http"; +import { tmpdir } from "node:os"; +import { dirname, join } from "node:path"; +import test from "node:test"; +import { fileURLToPath } from "node:url"; +import { computePushPlan, pushBlockingFindings } from "../src/push-plan.ts"; +import type { LoadedResources, ResourceType, StateFile } from "../src/types.ts"; +import { VALID_RESOURCE_TYPES } from "../src/types.ts"; + +process.argv = ["node", "test", "preflight"]; +process.env.VAPI_TOKEN = "fake"; +const { validateResources } = await import("../src/validate.ts"); + +const ROOT = join(dirname(fileURLToPath(import.meta.url)), ".."); +const UUID = "a0000000-0000-4000-8000-000000000001"; +const BAD = "An assistant name that is far longer than forty characters"; + +function loaded( + files: Partial>> = {}, +): LoadedResources { + const list = (type: ResourceType) => + Object.entries(files[type] ?? {}).map(([resourceId, data]) => ({ + resourceId, + data: data as Record, + filePath: `resources/preflight/${type}/${resourceId}.yml`, + })); + return { + tools: list("tools"), + structuredOutputs: list("structuredOutputs"), + assistants: list("assistants"), + squads: list("squads"), + personalities: list("personalities"), + scenarios: list("scenarios"), + simulations: list("simulations"), + simulationSuites: list("simulationSuites"), + evals: list("evals"), + }; +} + +function state(): StateFile { + return { + tools: {}, + structuredOutputs: {}, + assistants: {}, + squads: {}, + personalities: {}, + scenarios: {}, + simulations: {}, + simulationSuites: {}, + evals: {}, + credentials: { unused: { uuid: UUID } }, + }; +} + +test("plan: empty selection, all resource types, and existing dependencies", () => { + assert.deepEqual([...computePushPlan(loaded(), loaded(), state())], []); + const all = loaded( + Object.fromEntries( + VALID_RESOURCE_TYPES.map((type) => [type, { item: {} }]), + ), + ); + assert.equal( + computePushPlan(all, all, state()).size, + VALID_RESOURCE_TYPES.length, + ); + const local = loaded({ + assistants: { + bot: { + model: { toolIds: ["tool", UUID] }, + artifactPlan: { structuredOutputIds: ["output"] }, + }, + }, + tools: { tool: {} }, + structuredOutputs: { output: {} }, + }); + const selected = { ...loaded(), assistants: local.assistants }; + assert.deepEqual([...computePushPlan(selected, local, state())].sort(), [ + "assistants:bot", + "structuredOutputs:output", + "tools:tool", + ]); + const tracked = state(); + tracked.tools.tool = { uuid: UUID }; + tracked.structuredOutputs.output = { uuid: UUID }; + assert.deepEqual( + [...computePushPlan(selected, local, tracked)], + ["assistants:bot"], + ); +}); + +test("plan: squad auto-applied assistant errors block, including existing assistants updated for new dependencies", () => { + const local = loaded({ + squads: { team: { members: [{ assistantId: "bot" }] } }, + assistants: { bot: { name: BAD, model: { toolIds: ["tool"] } } }, + tools: { tool: {} }, + }); + const selected = { ...loaded(), squads: local.squads }; + const tracked = state(); + for (const existing of [false, true]) { + if (existing) tracked.assistants.bot = { uuid: UUID }; + const plan = computePushPlan(selected, local, tracked); + assert.ok(plan.has("assistants:bot")); + assert.ok(plan.has("tools:tool")); + assert.equal( + pushBlockingFindings(validateResources(local), plan, false).length, + 1, + ); + } + tracked.tools.tool = { uuid: UUID }; + assert.deepEqual( + [...computePushPlan(selected, local, tracked)], + ["squads:team"], + ); + assert.equal( + pushBlockingFindings(validateResources(local), new Set(), true).length, + 1, + ); + assert.deepEqual(pushBlockingFindings([], new Set(), false), []); +}); + +async function runFixture(options: { + scoped?: boolean; + dryRun?: boolean; + credential?: boolean; + strict?: boolean; +}) { + const dir = mkdtempSync(join(tmpdir(), "push-preflight-")); + const mutations: string[] = []; + const good = { id: UUID, name: "Good" }; + const bad = { + id: "b0000000-0000-4000-8000-000000000002", + name: options.credential ? "Bad" : BAD, + ...(options.credential + ? { server: { url: "https://example.com", credentialId: "missing" } } + : {}), + }; + const server = createServer((req, res) => { + const method = req.method ?? "GET"; + if (["POST", "PATCH", "DELETE"].includes(method)) + mutations.push(`${method} ${req.url}`); + res.setHeader("Content-Type", "application/json"); + const path = req.url?.split("?")[0]; + res.end( + JSON.stringify( + path === "/assistant" + ? [good, bad] + : path?.startsWith("/assistant/") + ? good + : [], + ), + ); + }); + await new Promise((resolve) => server.listen(0, "127.0.0.1", resolve)); + try { + const addr = server.address(); + assert.ok(addr && typeof addr === "object"); + cpSync(join(ROOT, "src"), join(dir, "src"), { recursive: true }); + cpSync(join(ROOT, "package.json"), join(dir, "package.json")); + symlinkSync(join(ROOT, "node_modules"), join(dir, "node_modules"), "dir"); + const folder = join(dir, "resources/preflight/assistants"); + mkdirSync(folder, { recursive: true }); + writeFileSync(join(folder, "good.yml"), "name: Good\n"); + writeFileSync( + join(folder, "zbad.yml"), + `name: ${bad.name}\n${options.credential ? "server:\n url: https://example.com\n credentialId: missing\n" : ""}`, + ); + const initial = state(); + initial.assistants.good = { uuid: good.id }; + initial.assistants.zbad = { uuid: bad.id }; + writeFileSync( + join(dir, ".vapi-state.preflight.json"), + JSON.stringify(initial), + ); + writeFileSync( + join(dir, ".env.preflight"), + `VAPI_TOKEN=fake\nVAPI_BASE_URL=http://127.0.0.1:${addr.port}\n`, + ); + const args = ["--import", "tsx", "src/push.ts", "preflight"]; + if (options.scoped) args.push("resources/preflight/assistants/good.yml"); + if (options.dryRun) args.push("--dry-run"); + if (options.strict) args.push("--strict"); + const result = await new Promise<{ code: number | null; output: string }>( + (resolve, reject) => { + const child = spawn(process.execPath, args, { + cwd: dir, + env: { + ...process.env, + VAPI_TOKEN: "fake", + VAPI_BASE_URL: `http://127.0.0.1:${addr.port}`, + }, + }); + let output = ""; + child.stdout.on("data", (chunk) => { + output += chunk; + }); + child.stderr.on("data", (chunk) => { + output += chunk; + }); + child.on("error", reject); + child.on("close", (code) => resolve({ code, output })); + }, + ); + return { ...result, mutations }; + } finally { + await new Promise((resolve, reject) => + server.close((err) => (err ? reject(err) : resolve())), + ); + rmSync(dir, { recursive: true, force: true }); + } +} + +for (const [label, options] of [ + ["full push", {}], + ["dry-run", { dryRun: true }], + ["unresolved credential", { credential: true }], + ["strict scoped push", { scoped: true, strict: true }], +] as const) { + test(`preflight blocks ${label} before any mutation`, async () => { + const result = await runFixture(options); + assert.equal(result.code, 1, result.output); + assert.match(result.output, /Nothing was written to the remote API/); + assert.deepEqual(result.mutations, []); + if ("credential" in options) + assert.match(result.output, /unresolved-credential/); + }); +} + +test("scoped valid push writes despite unrelated validation error", async () => { + const result = await runFixture({ scoped: true }); + assert.equal(result.code, 0, result.output); + assert.deepEqual(result.mutations, [`PATCH /assistant/${UUID}`]); + assert.match(result.output, /outside the push plan/); +});