From 1563a8de28d42950024658f4c8b33c9eced07c87 Mon Sep 17 00:00:00 2001 From: dhruva Date: Fri, 20 Nov 2026 23:20:40 +0000 Subject: [PATCH 1/4] feat(validate): warn on undefined squad variables Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- README.md | 2 +- docs/guides/blocks.md | 11 ++ docs/guides/commands.md | 2 +- improvements.md | 15 ++ src/push.ts | 6 +- src/validate-cmd.ts | 7 +- src/validate-variables.ts | 214 ++++++++++++++++++++++ src/validate.ts | 2 + tests/validate-variables.test.ts | 297 +++++++++++++++++++++++++++++++ 9 files changed, 552 insertions(+), 4 deletions(-) create mode 100644 src/validate-variables.ts create mode 100644 tests/validate-variables.test.ts diff --git a/README.md b/README.md index fd05170..3a0b78f 100644 --- a/README.md +++ b/README.md @@ -210,7 +210,7 @@ More detail: [How the engine works](docs/guides/how-it-works.md). | Command | What it does | | --- | --- | | `npm run setup` | Connect an org: create `.env.` and `resources//` | -| `npm run validate -- ` | Check resource files offline before deploying | +| `npm run validate -- [--strict]` | Check resource files offline, including undefined squad variables (`variable-undefined`); `--strict` blocks on that warning | | `npm run apply -- ` | Deploy: pull, merge, then push (the default) | | `npm run pull -- ` | Sync platform changes down without overwriting local edits | | `npm run push -- ` | Push without pulling first (prefer `apply`) | diff --git a/docs/guides/blocks.md b/docs/guides/blocks.md index bfc1dd0..8500ddb 100644 --- a/docs/guides/blocks.md +++ b/docs/guides/blocks.md @@ -4,6 +4,17 @@ A block is a piece of prompt text or config that lives in one file and is used by many resources. Change the block and every assistant and squad that includes it changes with it, in every org. +`validate` checks Liquid variables after expanding blocks. For each squad member, +`variable-undefined` warns about root names missing from the member's +`assistantOverrides.variableValues` and the squad's `membersOverrides.variableValues`. +It checks system prompts, first/voicemail/end-call messages, and attached tools, +including `tools:append`. Vapi built-ins, squad tool/handoff extraction schema +properties and alias keys, and Liquid assign/capture/loop locals count as defined. +Raw blocks and tool code are ignored. Standalone assistants are not checked: +their variables can be supplied at call time. This is a static check, not proof +that an extraction tool runs before a variable is used. Use `--strict` with +validate or push to make missing squad variables blocking. + There are two kinds: | Kind | File | Included with | Use it for | diff --git a/docs/guides/commands.md b/docs/guides/commands.md index 713f3c5..71c699b 100644 --- a/docs/guides/commands.md +++ b/docs/guides/commands.md @@ -10,7 +10,7 @@ The other commands are direct only. | Command | Usage | What it does | | --- | --- | --- | | `npm run setup` | `npm run setup [-- ]` | Connect an org: creates `.env.` and `resources//`. | -| `npm run validate` | `npm run validate -- ` | Check resource files offline: API shape rules, and that every reference names a file or a state entry. `apply` runs it first. On GitHub Actions, findings are also shown on the files in the pull request. | +| `npm run validate` | `npm run validate -- [--strict]` | Check resource files offline: API shape rules, references, and undefined squad variables. `variable-undefined` warns once per squad member, listing missing root names and where to define them; `--strict` makes this warning blocking in validate and push. `apply` runs validation first. On GitHub Actions, findings are also shown on the files in the pull request. | | `npm run apply` | `npm run apply -- [types or paths]` | **The default deploy:** pull, merge, then push. See [workflows](workflows.md). | | `npm run pull` | `npm run pull -- [--force] [--bootstrap]` | Sync platform changes down; never overwrites local edits unless `--force`. | | `npm run push` | `npm run push -- [--dry-run] [--strict]` | Push without pulling first. Prefer `apply`. `--strict` aborts before any API call if validation finds an error. | diff --git a/improvements.md b/improvements.md index eb8d0b9..c01e21c 100644 --- a/improvements.md +++ b/improvements.md @@ -2334,6 +2334,21 @@ have a bootstrap pull leave existing baselines alone. --- +## 43. Missing practice variables reached live prompts literally + +Shared assistants can run in many practice squads, each supplying practice +details through variableValues. A missing value stayed as `{{name}}` on a call. +`variable-undefined` now checks each squad member's expanded prompts and attached +tools against squad/member values, Vapi built-ins and runtime assignments. It +warns by default and blocks with `--strict`. Standalone assistants remain +unchecked because their values may arrive when a call is created. + +### Status + +**Implemented.** Covered by `tests/validate-variables.test.ts`. + +--- + ## Out of scope (intentionally not improvements) - **State file is identity-only and not git-ignored.** It's intentionally diff --git a/src/push.ts b/src/push.ts index e657e33..b5bdf7e 100644 --- a/src/push.ts +++ b/src/push.ts @@ -1837,7 +1837,11 @@ async function main(): Promise { } else { console.log(" ✅ No validation issues."); } - const errorCount = findings.filter((f) => f.severity === "error").length; + const errorCount = findings.filter( + (f) => + f.severity === "error" || + (STRICT_VALIDATION && f.rule === "variable-undefined"), + ).length; if (errorCount > 0) { if (STRICT_VALIDATION) { console.error( diff --git a/src/validate-cmd.ts b/src/validate-cmd.ts index 681c630..a0643ba 100644 --- a/src/validate-cmd.ts +++ b/src/validate-cmd.ts @@ -12,6 +12,7 @@ import { BASE_DIR, loadIgnorePatterns, RESOURCES_DIR, + STRICT_VALIDATION, VAPI_ENV, } from "./config.ts"; import { @@ -96,7 +97,11 @@ async function main(): Promise { } } - const errorCount = findings.filter((f) => f.severity === "error").length; + const errorCount = findings.filter( + (f) => + f.severity === "error" || + (STRICT_VALIDATION && f.rule === "variable-undefined"), + ).length; if (errorCount > 0) { console.error( `❌ Validation failed with ${errorCount} error(s). Fix the issues above before pushing.`, diff --git a/src/validate-variables.ts b/src/validate-variables.ts new file mode 100644 index 0000000..7d8f75c --- /dev/null +++ b/src/validate-variables.ts @@ -0,0 +1,214 @@ +import type { LoadedResources } from "./types.ts"; +import type { ValidationFinding } from "./validate.ts"; + +type Data = Record; +const record = (value: unknown): Data => + value && typeof value === "object" && !Array.isArray(value) + ? (value as Data) + : {}; +const list = (value: unknown): unknown[] => (Array.isArray(value) ? value : []); + +// Vapi: libs/core/src/util/mergeUtil/{assistantDefaultOverridesBuild,variableUtil}.ts +// (assistantDefaultOverridesBuild and timeVariablesGet), via handoffVariableBagBuild. +const BUILT_INS = [ + "customer", + "phoneNumber", + "transport", + "call", + "now", + "currentDateTime", + "date", + "time", + "year", + "month", + "day", +]; +const LITERALS = new Set([ + "true", + "false", + "nil", + "null", + "empty", + "blank", + "and", + "or", + "contains", +]); + +function references(text: string): Set { + const found = new Set(); + const locals = new Set(); + const loops: Set[] = []; + const clean = text.replace( + /{%-?\s*(raw|comment)\s*-?%}[\s\S]*?{%-?\s*end\1\s*-?%}/g, + "", + ); + const expression = (value: string) => { + const bare = value.replace( + /"[^"\\]*(?:\\.[^"\\]*)*"|'[^'\\]*(?:\\.[^'\\]*)*'/g, + "''", + ); + for (const match of bare.matchAll( + /(? scope.has(name)) + ) + found.add(name); + } + }; + for (const token of clean.matchAll( + /{{-?([\s\S]*?)-?}}|{%-?([\s\S]*?)-?%}/g, + )) { + if (token[1] !== undefined) { + expression(token[1]); + continue; + } + const tag = token[2]!.trim(); + const assign = /^(assign|capture)\s+([\w-]+)(?:\s*=\s*([\s\S]*))?/.exec( + tag, + ); + if (assign) { + if (assign[3]) expression(assign[3]); + locals.add(assign[2]!); + } else { + const loop = /^for\s+([\w-]+)\s+in\s+([\s\S]*)/.exec(tag); + if (loop) { + expression(loop[2]!); + loops.push(new Set([loop[1]!, "forloop"])); + } else if (/^endfor\b/.test(tag)) loops.pop(); + else if (/^(if|unless|elsif|case|when|echo)\s/.test(tag)) + expression(tag.replace(/^\w+\s+/, "")); + } + } + return found; +} + +function strings(value: unknown): string[] { + if (typeof value === "string") return [value]; + if (Array.isArray(value)) return value.flatMap(strings); + return Object.values(record(value)).flatMap(strings); +} + +function merge(base: unknown, overrides: unknown): Data { + const result = { ...record(base) }; + for (const [key, value] of Object.entries(record(overrides))) { + result[key] = Object.keys(record(value)).length + ? merge(result[key], value) + : value; + } + return result; +} + +function extractionNames(value: unknown, names: Set): void { + if (Array.isArray(value)) { + for (const child of value) extractionNames(child, names); + return; + } + const data = record(value); + const plan = record(data.variableExtractionPlan); + for (const name of Object.keys(record(record(plan.schema).properties))) + names.add(name); + for (const alias of list(plan.aliases)) { + const key = record(alias).key; + if (typeof key === "string") names.add(key); + } + for (const [key, child] of Object.entries(data)) { + if (key !== "variableExtractionPlan") extractionNames(child, names); + } +} + +function assistantTexts(assistant: Data): string[] { + const model = record(assistant.model); + return [ + assistant.firstMessage, + assistant.voicemailMessage, + assistant.endCallMessage, + model.systemPrompt, + ...list(model.messages) + .filter((message) => record(message).role === "system") + .map((message) => record(message).content), + ].flatMap(strings); +} + +function toolTexts(tool: unknown): string[] { + // Code is executed, not Liquid-rendered; extraction aliases have their own result bag. + const { code: _code, variableExtractionPlan: _plan, ...data } = record(tool); + return Object.entries(data).flatMap(([key, value]) => { + if (key === "destinations") return list(value).flatMap(toolTexts); + return strings(value); + }); +} + +export function validateVariables( + resources: LoadedResources, +): ValidationFinding[] { + const assistants = new Map( + resources.assistants.map((file) => [file.resourceId, file.data]), + ); + const tools = new Map( + resources.tools.map((file) => [file.resourceId, file.data]), + ); + const findings: ValidationFinding[] = []; + for (const squadFile of resources.squads) { + const squad = record(squadFile.data); + const shared = record(squad.membersOverrides); + const members = list(squad.members).map((value, index) => { + const member = record(value); + const own = record(member.assistantOverrides); + const overrides = { ...own, ...shared }; + const base = + typeof member.assistantId === "string" + ? assistants.get(member.assistantId) + : member.assistant; + const assistant = merge(base, overrides); + const model = record(assistant.model); + const attached = [ + ...list(model.tools), + ...list(model.toolIds).flatMap((id) => + typeof id === "string" && tools.has(id) ? [tools.get(id)] : [], + ), + ...list(own["tools:append"]), + ...list(shared["tools:append"]), + ]; + return { member, own, assistant, attached, index }; + }); + const runtime = new Set(BUILT_INS); + for (const member of members) extractionNames(member.attached, runtime); + for (const { member, own, assistant, attached, index } of members) { + const defined = new Set([ + ...runtime, + ...Object.keys(record(own.variableValues)), + ...Object.keys(record(shared.variableValues)), + ]); + const missing = new Set(); + for (const text of [ + ...assistantTexts(assistant), + ...attached.flatMap(toolTexts), + ]) { + for (const name of references(text)) + if (!defined.has(name)) missing.add(name); + } + if (missing.size === 0) continue; + const label = + typeof member.assistantId === "string" + ? member.assistantId + : String(record(member.assistant).name ?? index); + findings.push({ + severity: "warn", + type: "squads", + resourceId: squadFile.resourceId, + rule: "variable-undefined", + fieldPath: `members[${index}]`, + message: `squad "${squadFile.resourceId}" member "${label}" uses undefined variables: ${[...missing].sort().join(", ")}. Set them in membersOverrides.variableValues or this member's assistantOverrides.variableValues.`, + }); + } + } + return findings; +} diff --git a/src/validate.ts b/src/validate.ts index 2d6a7eb..7986f77 100644 --- a/src/validate.ts +++ b/src/validate.ts @@ -18,6 +18,7 @@ import { matchesIgnore } from "./config.ts"; import { RESOURCE_TYPES_WITH_REFS, referencesCollect } from "./resolver.ts"; import { FOLDER_MAP } from "./resources.ts"; import type { LoadedResources, ResourceFile, ResourceType } from "./types.ts"; +import { validateVariables } from "./validate-variables.ts"; export type ValidationSeverity = "warn" | "error"; @@ -545,6 +546,7 @@ export function validateResources( ...checkMaxTokensFloor(resources), ...checkVoiceSchemas(resources), ...checkIgnoredSharing(resources), + ...validateVariables(resources), ]; } diff --git a/tests/validate-variables.test.ts b/tests/validate-variables.test.ts new file mode 100644 index 0000000..5e43faf --- /dev/null +++ b/tests/validate-variables.test.ts @@ -0,0 +1,297 @@ +import "./engine-sandbox.ts"; +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { rmSync } from "node:fs"; +import { join } from "node:path"; +import test from "node:test"; +import { blockLayers, blocksExpand } from "../src/blocks.ts"; +import type { LoadedResources, ResourceFile } from "../src/types.ts"; +import { engineRepoCreate, repoCreate } from "./blocks-helpers.ts"; + +process.argv = ["node", "test", "test-fixture-org"]; +process.env.VAPI_TOKEN ||= "test-token-not-used"; +const { validateResources } = await import("../src/validate.ts"); + +const file = ( + resourceId: string, + data: Record, +): ResourceFile => ({ resourceId, data, filePath: `/fake/${resourceId}.yml` }); +const prompt = (content: string) => ({ + model: { messages: [{ role: "system", content }] }, +}); +function resources( + text = "{{practice}}", + overrides: object = {}, +): LoadedResources { + return { + assistants: [file("desk", prompt(text))], + squads: [ + file("north", { members: [{ assistantId: "desk" }], ...overrides }), + ], + tools: [], + structuredOutputs: [], + personalities: [], + scenarios: [], + simulations: [], + simulationSuites: [], + evals: [], + }; +} +const findings = (loaded: LoadedResources) => + validateResources(loaded).filter((f) => f.rule === "variable-undefined"); + +test("variables defined at squad or member level, including empty values", () => { + assert.deepEqual( + findings( + resources(undefined, { + membersOverrides: { variableValues: { practice: "" } }, + }), + ), + [], + ); + assert.deepEqual( + findings( + resources(undefined, { + members: [ + { + assistantId: "desk", + assistantOverrides: { variableValues: { practice: "North" } }, + }, + ], + }), + ), + [], + ); +}); + +test("built-in roots, property access, filters and literals", () => { + assert.deepEqual( + findings( + resources( + "{{customer.number | default: 'unknown'}} {{phoneNumber.number}} {{transport}} {{call.id}} {{now}} {{currentDateTime}} {{date}} {{time}} {{year}} {{month}} {{day}}", + ), + ), + [], + ); +}); + +test("one finding per member, isolated between two practice squads", () => { + const loaded = resources("{{practice}} {{mailbox}} {{practice}}"); + loaded.squads.push( + file("south", { + members: [{ assistantId: "desk" }], + membersOverrides: { + variableValues: { practice: "South", mailbox: "hello" }, + }, + }), + ); + const result = findings(loaded); + assert.equal(result.length, 1); + assert.equal(result[0]!.severity, "warn"); + assert.equal(result[0]!.resourceId, "north"); + assert.match( + result[0]!.message, + /member "desk".*mailbox, practice.*membersOverrides.variableValues.*assistantOverrides.variableValues/, + ); +}); + +test("extraction schemas and aliases on tools and handoff destinations in any member", () => { + const loaded = resources("{{patient}} {{alias}} {{destination}}"); + loaded.squads[0] = file("north", { + members: [ + { assistantId: "desk" }, + { + assistant: { + model: { + tools: [ + { + variableExtractionPlan: { + schema: { properties: { patient: { type: "string" } } }, + aliases: [{ key: "alias", value: "{{response.name}}" }], + }, + destinations: [ + { + variableExtractionPlan: { + schema: { properties: { destination: {} } }, + }, + }, + ], + }, + ], + }, + }, + }, + ], + }); + assert.deepEqual(findings(loaded), []); +}); + +test("Liquid assigns, capture and loop locals are local to each text", () => { + assert.deepEqual( + findings( + resources( + "{% assign name = 'North' %}{{name}} {% capture greeting %}Hello{% endcapture %}{{greeting}} {% for x in customer.items %}{{x.name}}{{forloop.index}}{% endfor %}", + ), + ), + [], + ); + const result = findings( + resources( + "{% assign name = missing %}{{name}} {% for x in customer.items %}{{x}}{% endfor %}{{x}}", + ), + ); + assert.match(result[0]!.message, /missing, x/); + const loaded = resources("{% assign local = 'yes' %}{{local}}"); + loaded.assistants[0] = file("desk", { + ...prompt("{% assign local = 'yes' %}{{local}}"), + firstMessage: "{{local}}", + }); + assert.match(findings(loaded)[0]!.message, /local/); +}); + +test("raw, whitespace-control raw and comment blocks are ignored", () => { + assert.deepEqual( + findings( + resources( + "{% raw %}{{missing}}{% endraw %}{%- raw -%}{{other}}{%- endraw -%}{% comment %}{{hidden}}{% endcomment %}", + ), + ), + [], + ); +}); + +test("tool file destinations and member/squad appended tools are checked", () => { + const loaded = resources("Hello", { + membersOverrides: { + "tools:append": [{ destinations: [{ number: "{{shared}}" }] }], + }, + members: [ + { + assistantId: "desk", + assistantOverrides: { + "tools:append": [{ function: { description: "{{own}}" } }], + }, + }, + ], + }); + loaded.assistants[0] = file("desk", { model: { toolIds: ["transfer"] } }); + loaded.tools = [ + file("transfer", { + type: "transferCall", + destinations: [{ number: "{{handover_number}}" }], + }), + ]; + assert.match(findings(loaded)[0]!.message, /handover_number, own, shared/); +}); + +test("inline assistant messages and effective override prompts are checked", () => { + const loaded = resources("{{replaced}}", { + members: [ + { + assistant: { + ...prompt("{{old}}"), + firstMessage: "{{first}}", + voicemailMessage: "{{voicemail}}", + endCallMessage: "{{end}}", + }, + assistantOverrides: prompt("{{override}}"), + }, + ], + }); + assert.match(findings(loaded)[0]!.message, /end, first, override, voicemail/); + assert.doesNotMatch(findings(loaded)[0]!.message, /old|replaced/); +}); + +test("standalone assistants, unused tools and non-rendered code are not checked", () => { + const loaded = resources("Hello"); + loaded.assistants.push(file("standalone", prompt("{{call_time}}"))); + loaded.tools.push( + file("unused", { function: { description: "{{unused}}" } }), + ); + loaded.squads[0] = file("north", { + members: [ + { + assistantId: "desk", + assistantOverrides: { + "tools:append": [{ type: "code", code: "return '{{javascript}}'" }], + }, + }, + ], + }); + assert.deepEqual(findings(loaded), []); + assert.deepEqual(findings({ ...loaded, squads: [] }), []); +}); + +test("variables introduced by expanded blocks are checked", () => { + const root = repoCreate({ + "templates/blocks/practice.md": "Welcome to {{practice}}", + }); + try { + const loaded = resources(); + loaded.assistants[0] = file( + "desk", + blocksExpand( + prompt("@include practice"), + blockLayers(join(root, "resources", "north")), + ).data, + ); + assert.match(findings(loaded)[0]!.message, /practice/); + } finally { + rmSync(root, { recursive: true, force: true }); + } +}); + +test("validate warns normally and blocks undefined variables with --strict", () => { + const root = engineRepoCreate({ + "resources/north/assistants/desk.yml": + "name: Desk\nmodel:\n provider: openai\n model: gpt-4o\n messages:\n - role: system\n content: 'Welcome to {{practice}}'\n", + "resources/north/squads/practice.yml": + "name: Practice\nmembers:\n - assistantId: desk\n", + }); + try { + for (const strict of [false, true]) { + const result = spawnSync( + process.execPath, + [ + "--import", + "tsx", + "src/validate-cmd.ts", + "north", + ...(strict ? ["--strict"] : []), + ], + { + cwd: root, + env: { + ...process.env, + VAPI_GITOPS_ROOT: root, + VAPI_TOKEN: "test-token-not-used", + VAPI_PRIVATE_API_KEY: "test-token-not-used", + }, + encoding: "utf8", + }, + ); + assert.match(result.stdout, /variable-undefined/); + assert.equal( + result.status, + strict ? 1 : 0, + result.stdout + result.stderr, + ); + } + } finally { + rmSync(root, { recursive: true, force: true }); + } +}); + +test("empty or malformed optional fields do not crash validation", () => { + const loaded = resources("Hello", { + members: [ + null, + {}, + { + assistant: { model: { tools: [null], messages: null } }, + assistantOverrides: { variableValues: null }, + }, + ], + }); + assert.deepEqual(findings(loaded), []); +}); From 4995c5594c4d8aa24b7b791081fdde310c9e66b4 Mon Sep 17 00:00:00 2001 From: dhruva Date: Fri, 20 Nov 2026 23:31:18 +0000 Subject: [PATCH 2/4] fix(validate): reject Liquid transfer destination numbers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- README.md | 2 +- docs/guides/commands.md | 2 +- improvements.md | 21 +++++ src/validate-transfer-numbers.ts | 44 +++++++++ src/validate.ts | 2 + tests/validate-transfer-numbers.test.ts | 120 ++++++++++++++++++++++++ 6 files changed, 189 insertions(+), 2 deletions(-) create mode 100644 src/validate-transfer-numbers.ts create mode 100644 tests/validate-transfer-numbers.test.ts diff --git a/README.md b/README.md index 3a0b78f..232bc0d 100644 --- a/README.md +++ b/README.md @@ -210,7 +210,7 @@ More detail: [How the engine works](docs/guides/how-it-works.md). | Command | What it does | | --- | --- | | `npm run setup` | Connect an org: create `.env.` and `resources//` | -| `npm run validate -- [--strict]` | Check resource files offline, including undefined squad variables (`variable-undefined`); `--strict` blocks on that warning | +| `npm run validate -- [--strict]` | Check resource files offline, including undefined squad variables (`variable-undefined`); `--strict` blocks on that warning. `transfer-number-liquid` always errors on Liquid in transfer destination numbers; use per-practice literal numbers (see `examples/nhs-shared-squads`). | | `npm run apply -- ` | Deploy: pull, merge, then push (the default) | | `npm run pull -- ` | Sync platform changes down without overwriting local edits | | `npm run push -- ` | Push without pulling first (prefer `apply`) | diff --git a/docs/guides/commands.md b/docs/guides/commands.md index 71c699b..ff8315e 100644 --- a/docs/guides/commands.md +++ b/docs/guides/commands.md @@ -10,7 +10,7 @@ The other commands are direct only. | Command | Usage | What it does | | --- | --- | --- | | `npm run setup` | `npm run setup [-- ]` | Connect an org: creates `.env.` and `resources//`. | -| `npm run validate` | `npm run validate -- [--strict]` | Check resource files offline: API shape rules, references, and undefined squad variables. `variable-undefined` warns once per squad member, listing missing root names and where to define them; `--strict` makes this warning blocking in validate and push. `apply` runs validation first. On GitHub Actions, findings are also shown on the files in the pull request. | +| `npm run validate` | `npm run validate -- [--strict]` | Check resource files offline: API shape rules, references, and undefined squad variables. `variable-undefined` warns once per squad member, listing missing root names and where to define them; `--strict` makes this warning blocking in validate and push. `transfer-number-liquid` is always an error: the API rejects Liquid in transfer destination numbers, even with `numberE164CheckEnabled:false`; use per-practice literal numbers, e.g. inline in squad `membersOverrides['tools:append']` (see `examples/nhs-shared-squads`). Covers standalone tools, inline model tools and member/squad appended tools. `apply` runs validation first. On GitHub Actions, findings are also shown on the files in the pull request. | | `npm run apply` | `npm run apply -- [types or paths]` | **The default deploy:** pull, merge, then push. See [workflows](workflows.md). | | `npm run pull` | `npm run pull -- [--force] [--bootstrap]` | Sync platform changes down; never overwrites local edits unless `--force`. | | `npm run push` | `npm run push -- [--dry-run] [--strict]` | Push without pulling first. Prefer `apply`. `--strict` aborts before any API call if validation finds an error. | diff --git a/improvements.md b/improvements.md index c01e21c..8e916ac 100644 --- a/improvements.md +++ b/improvements.md @@ -2349,6 +2349,27 @@ unchecked because their values may arrive when a call is created. --- +## 44. Liquid transfer destination numbers passed offline validation + +### Problem + +The API rejects Liquid in transfer number destinations: the default E.164 +validator rejects it, and disabling that check still requires alphanumeric +characters with an optional leading plus. Runtime template support does not +make these tool payloads API-valid. A push could therefore fail partway through. + +### Current behavior and mitigation + +`transfer-number-liquid` reports an error before deployment for standalone and +inline transfer tools, including member/squad appended tools. Use literal +per-practice numbers in inline squad overrides; see `examples/nhs-shared-squads`. + +### Status + +**Implemented.** Covered by `tests/validate-transfer-numbers.test.ts`. + +--- + ## Out of scope (intentionally not improvements) - **State file is identity-only and not git-ignored.** It's intentionally diff --git a/src/validate-transfer-numbers.ts b/src/validate-transfer-numbers.ts new file mode 100644 index 0000000..2387c25 --- /dev/null +++ b/src/validate-transfer-numbers.ts @@ -0,0 +1,44 @@ +import type { LoadedResources } from "./types.ts"; +import type { ValidationFinding } from "./validate.ts"; + +export function validateTransferNumbers( + resources: LoadedResources, +): ValidationFinding[] { + const findings: ValidationFinding[] = []; + for (const type of ["tools", "assistants", "squads"] as const) { + for (const resource of resources[type]) { + const visit = (value: unknown, path: string): void => { + if (!value || typeof value !== "object") return; + if (Array.isArray(value)) { + value.forEach((entry, index) => visit(entry, `${path}[${index}]`)); + return; + } + const object = value as Record; + if ( + object.type === "transferCall" && + Array.isArray(object.destinations) + ) { + object.destinations.forEach((destination, index) => { + const number = destination?.number; + if (typeof number !== "string" || !/{{|{%/.test(number)) return; + // Vapi: libs/core/src/types/tool.types/transferCallTool.types.ts (TransferDestinationNumber). + findings.push({ + severity: "error", + type, + resourceId: resource.resourceId, + rule: "transfer-number-liquid", + fieldPath: `${path ? `${path}.` : ""}destinations[${index}].number`, + message: + "The API rejects Liquid in transfer numbers: E.164 validation applies, and numberE164CheckEnabled:false still requires ^\\+?[a-zA-Z0-9]+$. Use a per-practice literal number, e.g. an inline tool in the squad's membersOverrides['tools:append']; see examples/nhs-shared-squads.", + }); + }); + } + for (const [key, child] of Object.entries(object)) { + visit(child, path ? `${path}.${key}` : key); + } + }; + visit(resource.data, ""); + } + } + return findings; +} diff --git a/src/validate.ts b/src/validate.ts index 7986f77..842b340 100644 --- a/src/validate.ts +++ b/src/validate.ts @@ -18,6 +18,7 @@ import { matchesIgnore } from "./config.ts"; import { RESOURCE_TYPES_WITH_REFS, referencesCollect } from "./resolver.ts"; import { FOLDER_MAP } from "./resources.ts"; import type { LoadedResources, ResourceFile, ResourceType } from "./types.ts"; +import { validateTransferNumbers } from "./validate-transfer-numbers.ts"; import { validateVariables } from "./validate-variables.ts"; export type ValidationSeverity = "warn" | "error"; @@ -547,6 +548,7 @@ export function validateResources( ...checkVoiceSchemas(resources), ...checkIgnoredSharing(resources), ...validateVariables(resources), + ...validateTransferNumbers(resources), ]; } diff --git a/tests/validate-transfer-numbers.test.ts b/tests/validate-transfer-numbers.test.ts new file mode 100644 index 0000000..b4d4168 --- /dev/null +++ b/tests/validate-transfer-numbers.test.ts @@ -0,0 +1,120 @@ +import "./engine-sandbox.ts"; +import assert from "node:assert/strict"; +import test from "node:test"; +import type { LoadedResources, ResourceFile } from "../src/types.ts"; + +process.argv = ["node", "test", "test-fixture-org"]; +process.env.VAPI_TOKEN ||= "test-token-not-used"; +const { validateResources } = await import("../src/validate.ts"); + +const file = (data: Record): ResourceFile => ({ + resourceId: "example", + filePath: "/fake/example.yml", + data, +}); +const tool = (number: unknown = "{{handover_number}}", extra = {}) => ({ + type: "transferCall", + destinations: [{ type: "number", number, ...extra }], +}); +const loaded = (): LoadedResources => ({ + tools: [], + assistants: [], + squads: [], + structuredOutputs: [], + personalities: [], + scenarios: [], + simulations: [], + simulationSuites: [], + evals: [], +}); +const findings = (resources: LoadedResources) => + validateResources(resources).filter( + (f) => f.rule === "transfer-number-liquid", + ); + +for (const placement of [ + "standalone", + "model", + "inline-member", + "member", + "squad", +] as const) { + test(`Liquid transfer number rejected in ${placement}`, () => { + const resources = loaded(); + let type: "tools" | "assistants" | "squads" = "squads"; + let fieldPath: string; + if (placement === "standalone") { + type = "tools"; + resources.tools = [file(tool())]; + fieldPath = "destinations[0].number"; + } else if (placement === "model") { + type = "assistants"; + resources.assistants = [file({ model: { tools: [tool()] } })]; + fieldPath = "model.tools[0].destinations[0].number"; + } else if (placement === "inline-member") { + resources.squads = [ + file({ members: [{ assistant: { model: { tools: [tool()] } } }] }), + ]; + fieldPath = "members[0].assistant.model.tools[0].destinations[0].number"; + } else if (placement === "member") { + resources.squads = [ + file({ + members: [{ assistantOverrides: { "tools:append": [tool()] } }], + }), + ]; + fieldPath = + "members[0].assistantOverrides.tools:append[0].destinations[0].number"; + } else { + resources.squads = [ + file({ membersOverrides: { "tools:append": [tool()] } }), + ]; + fieldPath = "membersOverrides.tools:append[0].destinations[0].number"; + } + const result = findings(resources); + assert.equal(result.length, 1); + assert.deepEqual(result[0], { + severity: "error", + type, + resourceId: "example", + rule: "transfer-number-liquid", + fieldPath, + message: + "The API rejects Liquid in transfer numbers: E.164 validation applies, and numberE164CheckEnabled:false still requires ^\\+?[a-zA-Z0-9]+$. Use a per-practice literal number, e.g. an inline tool in the squad's membersOverrides['tools:append']; see examples/nhs-shared-squads.", + }); + }); +} + +test("Liquid tags and disabled E.164 checks still produce errors", () => { + const resources = loaded(); + resources.tools = [ + file( + tool("{% assign number = '123' %}", { numberE164CheckEnabled: false }), + ), + ]; + assert.equal(findings(resources).length, 1); +}); + +test("literal, empty and malformed destinations do not trigger the Liquid rule", () => { + const resources = loaded(); + resources.tools = [ + file(tool("+442079460101")), + file(tool(null)), + file({ type: "transferCall", destinations: [null, {}, "text"] }), + file({ type: "transferCall" }), + ]; + resources.assistants = [file({ model: { tools: [tool("+442079460201")] } })]; + resources.squads = [ + file({ membersOverrides: { "tools:append": [tool("+442079460202")] } }), + ]; + assert.deepEqual(findings(resources), []); + assert.deepEqual(findings(loaded()), []); +}); + +test("other tool types and Liquid descriptions are not transfer numbers", () => { + const resources = loaded(); + resources.tools = [ + file({ ...tool(), type: "handoff" }), + file({ ...tool("+442079460101"), description: "{{practice}}" }), + ]; + assert.deepEqual(findings(resources), []); +}); From 9ef7af97fde1826a170a9a55c1486929bfbb91bc Mon Sep 17 00:00:00 2001 From: dhruva Date: Sat, 21 Nov 2026 00:18:45 +0000 Subject: [PATCH 3/4] docs: explain squad variable and transfer number validation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- AGENTS.md | 5 +++++ docs/learnings/squads.md | 12 ++++++++++-- docs/learnings/tools.md | 8 ++++++++ 3 files changed, 23 insertions(+), 2 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index c8137f8..e0bcea6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -52,6 +52,9 @@ Safe to run without asking: `npm run validate`, `npm run audit`, plain `npm run pull` (it never overwrites local edits), `npm run push -- --dry-run`, `npm run build`, `npm test`. +- `variable-undefined`: set squad members' `{{vars}}` in squad/member `variableValues`; `--strict` blocks missing values. +- `transfer-number-liquid`: transfer destination numbers must be literal; put per-squad numbers in an inline `transferCall` in `membersOverrides["tools:append"]`. + **2. Never handle API keys.** Don't ask the human to paste a key into chat, don't print or read out `.env.*` files, never pass a key as a command-line argument, and never commit `.env.*`. Keys live in `.env.` (gitignored) @@ -150,6 +153,8 @@ precisely. never ask for a real key just to validate. Don't weaken the check or the workflow to get past it, and never edit the state file, `.vapi-ignore` or `state//org.json` to make a reference resolve: fix the name, or pull. + - `variable-undefined`: set squad members' `{{vars}}` in squad/member `variableValues`; `--strict` blocks this warning in validate and push. + - `transfer-number-liquid`: use literal transfer numbers; append a per-squad inline `transferCall` in the squad's `membersOverrides["tools:append"]`. 4. **Build PR checks offline** if `vapi-checks.yml` exists: `npm run check -- --all --dry-run`. Fix anything it reports. 5. **Deploy only with a yes** (safety rule 1): `npm run apply -- `, or diff --git a/docs/learnings/squads.md b/docs/learnings/squads.md index 1202b85..ae239f6 100644 --- a/docs/learnings/squads.md +++ b/docs/learnings/squads.md @@ -156,6 +156,16 @@ For a one-off call using an existing `squadId`, the same shape can be passed as ## Override Merge Order +### Shared assistants speak unresolved practice variables + +**Problem:** A shared assistant says literal `{{name}}` instead of a practice's details. + +**Cause:** Undefined variables remain literal on the call; an explicit empty string renders as nothing. Override precedence is member → squad → call; later values win. + +**Fix:** Keep assistants shared and set each practice's values in `membersOverrides.variableValues` (or a member's `assistantOverrides.variableValues`). `variable-undefined` warns per squad member; `--strict` blocks it in validate and push. Transfer numbers must stay literal: see [Liquid destination numbers](tools.md#liquid-destination-numbers-fail-api-validation). + +### Layer precedence + The final assistant configuration is built by merging these layers in order: 1. Base assistant @@ -169,8 +179,6 @@ Later layers win on conflicts. `variableValues` from all layers are merged separ **Gotcha:** a `credentialIds` list anywhere in the squad narrows the whole call to the listed keys, for every member and every provider. A provider left off runs on Vapi's own key with no error. List every key the call needs, and let `npm run validate` check it: see [Credential pins](../guides/credential-pins.md). -**Gotcha:** a variable with no value is **left in the prompt as written**, not replaced and not an error. If a variable isn't in the merged `variableValues`, `{{ myVar }}` reaches the model as the literal text `{{ myVar }}`. A variable that is set to an empty string does render as nothing. So a squad that forgets a variable shows the braces to the model: check every `{{ … }}` in a prompt, and in any [block](../guides/blocks.md) it includes, has a value in each squad that uses it. - --- ## Squad-level post-call extraction via `membersOverrides` (multi-member squads) diff --git a/docs/learnings/tools.md b/docs/learnings/tools.md index 592fdec..ae76ca6 100644 --- a/docs/learnings/tools.md +++ b/docs/learnings/tools.md @@ -157,6 +157,14 @@ If the schema doesn't meet these requirements, `strict` may be silently disabled ## transferCall Tools +### Liquid destination numbers fail API validation + +**Problem:** A transfer destination such as `number: '{{handover_number}}'` is rejected even when the variable is defined. + +**Cause:** The API requires E.164 by default; `numberE164CheckEnabled: false` still enforces `^\+?[a-zA-Z0-9]+$`, which rejects Liquid syntax. + +**Fix:** Use literal numbers. For shared assistants, put per-squad numbers in an inline `transferCall` in the squad's `membersOverrides["tools:append"]`. `transfer-number-liquid` reports an error before deployment. See [shared squad variables](squads.md#shared-assistants-speak-unresolved-practice-variables) and the [NHS example](../../examples/nhs-shared-squads/). + ### Forwarding numbers from the assistant are auto-injected **What you might expect:** Only the `destinations` array on the tool matters. From 490fac1ba732c8456db2391819b0b962ffe6169c Mon Sep 17 00:00:00 2001 From: dhruva Date: Sat, 21 Nov 2026 00:21:05 +0000 Subject: [PATCH 4/4] docs: link NHS example across stacked PR branches Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- docs/learnings/tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/learnings/tools.md b/docs/learnings/tools.md index ae76ca6..d027ba0 100644 --- a/docs/learnings/tools.md +++ b/docs/learnings/tools.md @@ -163,7 +163,7 @@ If the schema doesn't meet these requirements, `strict` may be silently disabled **Cause:** The API requires E.164 by default; `numberE164CheckEnabled: false` still enforces `^\+?[a-zA-Z0-9]+$`, which rejects Liquid syntax. -**Fix:** Use literal numbers. For shared assistants, put per-squad numbers in an inline `transferCall` in the squad's `membersOverrides["tools:append"]`. `transfer-number-liquid` reports an error before deployment. See [shared squad variables](squads.md#shared-assistants-speak-unresolved-practice-variables) and the [NHS example](../../examples/nhs-shared-squads/). +**Fix:** Use literal numbers. For shared assistants, put per-squad numbers in an inline `transferCall` in the squad's `membersOverrides["tools:append"]`. `transfer-number-liquid` reports an error before deployment. See [shared squad variables](squads.md#shared-assistants-speak-unresolved-practice-variables) and the [NHS example (PR #100)](https://github.com/VapiAI/gitops/pull/100). ### Forwarding numbers from the assistant are auto-injected