From 1563a8de28d42950024658f4c8b33c9eced07c87 Mon Sep 17 00:00:00 2001 From: dhruva Date: Fri, 20 Nov 2026 23:20:40 +0000 Subject: [PATCH 1/7] feat(validate): warn on undefined squad variables Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- README.md | 2 +- docs/guides/blocks.md | 11 ++ docs/guides/commands.md | 2 +- improvements.md | 15 ++ src/push.ts | 6 +- src/validate-cmd.ts | 7 +- src/validate-variables.ts | 214 ++++++++++++++++++++++ src/validate.ts | 2 + tests/validate-variables.test.ts | 297 +++++++++++++++++++++++++++++++ 9 files changed, 552 insertions(+), 4 deletions(-) create mode 100644 src/validate-variables.ts create mode 100644 tests/validate-variables.test.ts diff --git a/README.md b/README.md index fd05170..3a0b78f 100644 --- a/README.md +++ b/README.md @@ -210,7 +210,7 @@ More detail: [How the engine works](docs/guides/how-it-works.md). | Command | What it does | | --- | --- | | `npm run setup` | Connect an org: create `.env.` and `resources//` | -| `npm run validate -- ` | Check resource files offline before deploying | +| `npm run validate -- [--strict]` | Check resource files offline, including undefined squad variables (`variable-undefined`); `--strict` blocks on that warning | | `npm run apply -- ` | Deploy: pull, merge, then push (the default) | | `npm run pull -- ` | Sync platform changes down without overwriting local edits | | `npm run push -- ` | Push without pulling first (prefer `apply`) | diff --git a/docs/guides/blocks.md b/docs/guides/blocks.md index bfc1dd0..8500ddb 100644 --- a/docs/guides/blocks.md +++ b/docs/guides/blocks.md @@ -4,6 +4,17 @@ A block is a piece of prompt text or config that lives in one file and is used by many resources. Change the block and every assistant and squad that includes it changes with it, in every org. +`validate` checks Liquid variables after expanding blocks. For each squad member, +`variable-undefined` warns about root names missing from the member's +`assistantOverrides.variableValues` and the squad's `membersOverrides.variableValues`. +It checks system prompts, first/voicemail/end-call messages, and attached tools, +including `tools:append`. Vapi built-ins, squad tool/handoff extraction schema +properties and alias keys, and Liquid assign/capture/loop locals count as defined. +Raw blocks and tool code are ignored. Standalone assistants are not checked: +their variables can be supplied at call time. This is a static check, not proof +that an extraction tool runs before a variable is used. Use `--strict` with +validate or push to make missing squad variables blocking. + There are two kinds: | Kind | File | Included with | Use it for | diff --git a/docs/guides/commands.md b/docs/guides/commands.md index 713f3c5..71c699b 100644 --- a/docs/guides/commands.md +++ b/docs/guides/commands.md @@ -10,7 +10,7 @@ The other commands are direct only. | Command | Usage | What it does | | --- | --- | --- | | `npm run setup` | `npm run setup [-- ]` | Connect an org: creates `.env.` and `resources//`. | -| `npm run validate` | `npm run validate -- ` | Check resource files offline: API shape rules, and that every reference names a file or a state entry. `apply` runs it first. On GitHub Actions, findings are also shown on the files in the pull request. | +| `npm run validate` | `npm run validate -- [--strict]` | Check resource files offline: API shape rules, references, and undefined squad variables. `variable-undefined` warns once per squad member, listing missing root names and where to define them; `--strict` makes this warning blocking in validate and push. `apply` runs validation first. On GitHub Actions, findings are also shown on the files in the pull request. | | `npm run apply` | `npm run apply -- [types or paths]` | **The default deploy:** pull, merge, then push. See [workflows](workflows.md). | | `npm run pull` | `npm run pull -- [--force] [--bootstrap]` | Sync platform changes down; never overwrites local edits unless `--force`. | | `npm run push` | `npm run push -- [--dry-run] [--strict]` | Push without pulling first. Prefer `apply`. `--strict` aborts before any API call if validation finds an error. | diff --git a/improvements.md b/improvements.md index eb8d0b9..c01e21c 100644 --- a/improvements.md +++ b/improvements.md @@ -2334,6 +2334,21 @@ have a bootstrap pull leave existing baselines alone. --- +## 43. Missing practice variables reached live prompts literally + +Shared assistants can run in many practice squads, each supplying practice +details through variableValues. A missing value stayed as `{{name}}` on a call. +`variable-undefined` now checks each squad member's expanded prompts and attached +tools against squad/member values, Vapi built-ins and runtime assignments. It +warns by default and blocks with `--strict`. Standalone assistants remain +unchecked because their values may arrive when a call is created. + +### Status + +**Implemented.** Covered by `tests/validate-variables.test.ts`. + +--- + ## Out of scope (intentionally not improvements) - **State file is identity-only and not git-ignored.** It's intentionally diff --git a/src/push.ts b/src/push.ts index e657e33..b5bdf7e 100644 --- a/src/push.ts +++ b/src/push.ts @@ -1837,7 +1837,11 @@ async function main(): Promise { } else { console.log(" ✅ No validation issues."); } - const errorCount = findings.filter((f) => f.severity === "error").length; + const errorCount = findings.filter( + (f) => + f.severity === "error" || + (STRICT_VALIDATION && f.rule === "variable-undefined"), + ).length; if (errorCount > 0) { if (STRICT_VALIDATION) { console.error( diff --git a/src/validate-cmd.ts b/src/validate-cmd.ts index 681c630..a0643ba 100644 --- a/src/validate-cmd.ts +++ b/src/validate-cmd.ts @@ -12,6 +12,7 @@ import { BASE_DIR, loadIgnorePatterns, RESOURCES_DIR, + STRICT_VALIDATION, VAPI_ENV, } from "./config.ts"; import { @@ -96,7 +97,11 @@ async function main(): Promise { } } - const errorCount = findings.filter((f) => f.severity === "error").length; + const errorCount = findings.filter( + (f) => + f.severity === "error" || + (STRICT_VALIDATION && f.rule === "variable-undefined"), + ).length; if (errorCount > 0) { console.error( `❌ Validation failed with ${errorCount} error(s). Fix the issues above before pushing.`, diff --git a/src/validate-variables.ts b/src/validate-variables.ts new file mode 100644 index 0000000..7d8f75c --- /dev/null +++ b/src/validate-variables.ts @@ -0,0 +1,214 @@ +import type { LoadedResources } from "./types.ts"; +import type { ValidationFinding } from "./validate.ts"; + +type Data = Record; +const record = (value: unknown): Data => + value && typeof value === "object" && !Array.isArray(value) + ? (value as Data) + : {}; +const list = (value: unknown): unknown[] => (Array.isArray(value) ? value : []); + +// Vapi: libs/core/src/util/mergeUtil/{assistantDefaultOverridesBuild,variableUtil}.ts +// (assistantDefaultOverridesBuild and timeVariablesGet), via handoffVariableBagBuild. +const BUILT_INS = [ + "customer", + "phoneNumber", + "transport", + "call", + "now", + "currentDateTime", + "date", + "time", + "year", + "month", + "day", +]; +const LITERALS = new Set([ + "true", + "false", + "nil", + "null", + "empty", + "blank", + "and", + "or", + "contains", +]); + +function references(text: string): Set { + const found = new Set(); + const locals = new Set(); + const loops: Set[] = []; + const clean = text.replace( + /{%-?\s*(raw|comment)\s*-?%}[\s\S]*?{%-?\s*end\1\s*-?%}/g, + "", + ); + const expression = (value: string) => { + const bare = value.replace( + /"[^"\\]*(?:\\.[^"\\]*)*"|'[^'\\]*(?:\\.[^'\\]*)*'/g, + "''", + ); + for (const match of bare.matchAll( + /(? scope.has(name)) + ) + found.add(name); + } + }; + for (const token of clean.matchAll( + /{{-?([\s\S]*?)-?}}|{%-?([\s\S]*?)-?%}/g, + )) { + if (token[1] !== undefined) { + expression(token[1]); + continue; + } + const tag = token[2]!.trim(); + const assign = /^(assign|capture)\s+([\w-]+)(?:\s*=\s*([\s\S]*))?/.exec( + tag, + ); + if (assign) { + if (assign[3]) expression(assign[3]); + locals.add(assign[2]!); + } else { + const loop = /^for\s+([\w-]+)\s+in\s+([\s\S]*)/.exec(tag); + if (loop) { + expression(loop[2]!); + loops.push(new Set([loop[1]!, "forloop"])); + } else if (/^endfor\b/.test(tag)) loops.pop(); + else if (/^(if|unless|elsif|case|when|echo)\s/.test(tag)) + expression(tag.replace(/^\w+\s+/, "")); + } + } + return found; +} + +function strings(value: unknown): string[] { + if (typeof value === "string") return [value]; + if (Array.isArray(value)) return value.flatMap(strings); + return Object.values(record(value)).flatMap(strings); +} + +function merge(base: unknown, overrides: unknown): Data { + const result = { ...record(base) }; + for (const [key, value] of Object.entries(record(overrides))) { + result[key] = Object.keys(record(value)).length + ? merge(result[key], value) + : value; + } + return result; +} + +function extractionNames(value: unknown, names: Set): void { + if (Array.isArray(value)) { + for (const child of value) extractionNames(child, names); + return; + } + const data = record(value); + const plan = record(data.variableExtractionPlan); + for (const name of Object.keys(record(record(plan.schema).properties))) + names.add(name); + for (const alias of list(plan.aliases)) { + const key = record(alias).key; + if (typeof key === "string") names.add(key); + } + for (const [key, child] of Object.entries(data)) { + if (key !== "variableExtractionPlan") extractionNames(child, names); + } +} + +function assistantTexts(assistant: Data): string[] { + const model = record(assistant.model); + return [ + assistant.firstMessage, + assistant.voicemailMessage, + assistant.endCallMessage, + model.systemPrompt, + ...list(model.messages) + .filter((message) => record(message).role === "system") + .map((message) => record(message).content), + ].flatMap(strings); +} + +function toolTexts(tool: unknown): string[] { + // Code is executed, not Liquid-rendered; extraction aliases have their own result bag. + const { code: _code, variableExtractionPlan: _plan, ...data } = record(tool); + return Object.entries(data).flatMap(([key, value]) => { + if (key === "destinations") return list(value).flatMap(toolTexts); + return strings(value); + }); +} + +export function validateVariables( + resources: LoadedResources, +): ValidationFinding[] { + const assistants = new Map( + resources.assistants.map((file) => [file.resourceId, file.data]), + ); + const tools = new Map( + resources.tools.map((file) => [file.resourceId, file.data]), + ); + const findings: ValidationFinding[] = []; + for (const squadFile of resources.squads) { + const squad = record(squadFile.data); + const shared = record(squad.membersOverrides); + const members = list(squad.members).map((value, index) => { + const member = record(value); + const own = record(member.assistantOverrides); + const overrides = { ...own, ...shared }; + const base = + typeof member.assistantId === "string" + ? assistants.get(member.assistantId) + : member.assistant; + const assistant = merge(base, overrides); + const model = record(assistant.model); + const attached = [ + ...list(model.tools), + ...list(model.toolIds).flatMap((id) => + typeof id === "string" && tools.has(id) ? [tools.get(id)] : [], + ), + ...list(own["tools:append"]), + ...list(shared["tools:append"]), + ]; + return { member, own, assistant, attached, index }; + }); + const runtime = new Set(BUILT_INS); + for (const member of members) extractionNames(member.attached, runtime); + for (const { member, own, assistant, attached, index } of members) { + const defined = new Set([ + ...runtime, + ...Object.keys(record(own.variableValues)), + ...Object.keys(record(shared.variableValues)), + ]); + const missing = new Set(); + for (const text of [ + ...assistantTexts(assistant), + ...attached.flatMap(toolTexts), + ]) { + for (const name of references(text)) + if (!defined.has(name)) missing.add(name); + } + if (missing.size === 0) continue; + const label = + typeof member.assistantId === "string" + ? member.assistantId + : String(record(member.assistant).name ?? index); + findings.push({ + severity: "warn", + type: "squads", + resourceId: squadFile.resourceId, + rule: "variable-undefined", + fieldPath: `members[${index}]`, + message: `squad "${squadFile.resourceId}" member "${label}" uses undefined variables: ${[...missing].sort().join(", ")}. Set them in membersOverrides.variableValues or this member's assistantOverrides.variableValues.`, + }); + } + } + return findings; +} diff --git a/src/validate.ts b/src/validate.ts index 2d6a7eb..7986f77 100644 --- a/src/validate.ts +++ b/src/validate.ts @@ -18,6 +18,7 @@ import { matchesIgnore } from "./config.ts"; import { RESOURCE_TYPES_WITH_REFS, referencesCollect } from "./resolver.ts"; import { FOLDER_MAP } from "./resources.ts"; import type { LoadedResources, ResourceFile, ResourceType } from "./types.ts"; +import { validateVariables } from "./validate-variables.ts"; export type ValidationSeverity = "warn" | "error"; @@ -545,6 +546,7 @@ export function validateResources( ...checkMaxTokensFloor(resources), ...checkVoiceSchemas(resources), ...checkIgnoredSharing(resources), + ...validateVariables(resources), ]; } diff --git a/tests/validate-variables.test.ts b/tests/validate-variables.test.ts new file mode 100644 index 0000000..5e43faf --- /dev/null +++ b/tests/validate-variables.test.ts @@ -0,0 +1,297 @@ +import "./engine-sandbox.ts"; +import assert from "node:assert/strict"; +import { spawnSync } from "node:child_process"; +import { rmSync } from "node:fs"; +import { join } from "node:path"; +import test from "node:test"; +import { blockLayers, blocksExpand } from "../src/blocks.ts"; +import type { LoadedResources, ResourceFile } from "../src/types.ts"; +import { engineRepoCreate, repoCreate } from "./blocks-helpers.ts"; + +process.argv = ["node", "test", "test-fixture-org"]; +process.env.VAPI_TOKEN ||= "test-token-not-used"; +const { validateResources } = await import("../src/validate.ts"); + +const file = ( + resourceId: string, + data: Record, +): ResourceFile => ({ resourceId, data, filePath: `/fake/${resourceId}.yml` }); +const prompt = (content: string) => ({ + model: { messages: [{ role: "system", content }] }, +}); +function resources( + text = "{{practice}}", + overrides: object = {}, +): LoadedResources { + return { + assistants: [file("desk", prompt(text))], + squads: [ + file("north", { members: [{ assistantId: "desk" }], ...overrides }), + ], + tools: [], + structuredOutputs: [], + personalities: [], + scenarios: [], + simulations: [], + simulationSuites: [], + evals: [], + }; +} +const findings = (loaded: LoadedResources) => + validateResources(loaded).filter((f) => f.rule === "variable-undefined"); + +test("variables defined at squad or member level, including empty values", () => { + assert.deepEqual( + findings( + resources(undefined, { + membersOverrides: { variableValues: { practice: "" } }, + }), + ), + [], + ); + assert.deepEqual( + findings( + resources(undefined, { + members: [ + { + assistantId: "desk", + assistantOverrides: { variableValues: { practice: "North" } }, + }, + ], + }), + ), + [], + ); +}); + +test("built-in roots, property access, filters and literals", () => { + assert.deepEqual( + findings( + resources( + "{{customer.number | default: 'unknown'}} {{phoneNumber.number}} {{transport}} {{call.id}} {{now}} {{currentDateTime}} {{date}} {{time}} {{year}} {{month}} {{day}}", + ), + ), + [], + ); +}); + +test("one finding per member, isolated between two practice squads", () => { + const loaded = resources("{{practice}} {{mailbox}} {{practice}}"); + loaded.squads.push( + file("south", { + members: [{ assistantId: "desk" }], + membersOverrides: { + variableValues: { practice: "South", mailbox: "hello" }, + }, + }), + ); + const result = findings(loaded); + assert.equal(result.length, 1); + assert.equal(result[0]!.severity, "warn"); + assert.equal(result[0]!.resourceId, "north"); + assert.match( + result[0]!.message, + /member "desk".*mailbox, practice.*membersOverrides.variableValues.*assistantOverrides.variableValues/, + ); +}); + +test("extraction schemas and aliases on tools and handoff destinations in any member", () => { + const loaded = resources("{{patient}} {{alias}} {{destination}}"); + loaded.squads[0] = file("north", { + members: [ + { assistantId: "desk" }, + { + assistant: { + model: { + tools: [ + { + variableExtractionPlan: { + schema: { properties: { patient: { type: "string" } } }, + aliases: [{ key: "alias", value: "{{response.name}}" }], + }, + destinations: [ + { + variableExtractionPlan: { + schema: { properties: { destination: {} } }, + }, + }, + ], + }, + ], + }, + }, + }, + ], + }); + assert.deepEqual(findings(loaded), []); +}); + +test("Liquid assigns, capture and loop locals are local to each text", () => { + assert.deepEqual( + findings( + resources( + "{% assign name = 'North' %}{{name}} {% capture greeting %}Hello{% endcapture %}{{greeting}} {% for x in customer.items %}{{x.name}}{{forloop.index}}{% endfor %}", + ), + ), + [], + ); + const result = findings( + resources( + "{% assign name = missing %}{{name}} {% for x in customer.items %}{{x}}{% endfor %}{{x}}", + ), + ); + assert.match(result[0]!.message, /missing, x/); + const loaded = resources("{% assign local = 'yes' %}{{local}}"); + loaded.assistants[0] = file("desk", { + ...prompt("{% assign local = 'yes' %}{{local}}"), + firstMessage: "{{local}}", + }); + assert.match(findings(loaded)[0]!.message, /local/); +}); + +test("raw, whitespace-control raw and comment blocks are ignored", () => { + assert.deepEqual( + findings( + resources( + "{% raw %}{{missing}}{% endraw %}{%- raw -%}{{other}}{%- endraw -%}{% comment %}{{hidden}}{% endcomment %}", + ), + ), + [], + ); +}); + +test("tool file destinations and member/squad appended tools are checked", () => { + const loaded = resources("Hello", { + membersOverrides: { + "tools:append": [{ destinations: [{ number: "{{shared}}" }] }], + }, + members: [ + { + assistantId: "desk", + assistantOverrides: { + "tools:append": [{ function: { description: "{{own}}" } }], + }, + }, + ], + }); + loaded.assistants[0] = file("desk", { model: { toolIds: ["transfer"] } }); + loaded.tools = [ + file("transfer", { + type: "transferCall", + destinations: [{ number: "{{handover_number}}" }], + }), + ]; + assert.match(findings(loaded)[0]!.message, /handover_number, own, shared/); +}); + +test("inline assistant messages and effective override prompts are checked", () => { + const loaded = resources("{{replaced}}", { + members: [ + { + assistant: { + ...prompt("{{old}}"), + firstMessage: "{{first}}", + voicemailMessage: "{{voicemail}}", + endCallMessage: "{{end}}", + }, + assistantOverrides: prompt("{{override}}"), + }, + ], + }); + assert.match(findings(loaded)[0]!.message, /end, first, override, voicemail/); + assert.doesNotMatch(findings(loaded)[0]!.message, /old|replaced/); +}); + +test("standalone assistants, unused tools and non-rendered code are not checked", () => { + const loaded = resources("Hello"); + loaded.assistants.push(file("standalone", prompt("{{call_time}}"))); + loaded.tools.push( + file("unused", { function: { description: "{{unused}}" } }), + ); + loaded.squads[0] = file("north", { + members: [ + { + assistantId: "desk", + assistantOverrides: { + "tools:append": [{ type: "code", code: "return '{{javascript}}'" }], + }, + }, + ], + }); + assert.deepEqual(findings(loaded), []); + assert.deepEqual(findings({ ...loaded, squads: [] }), []); +}); + +test("variables introduced by expanded blocks are checked", () => { + const root = repoCreate({ + "templates/blocks/practice.md": "Welcome to {{practice}}", + }); + try { + const loaded = resources(); + loaded.assistants[0] = file( + "desk", + blocksExpand( + prompt("@include practice"), + blockLayers(join(root, "resources", "north")), + ).data, + ); + assert.match(findings(loaded)[0]!.message, /practice/); + } finally { + rmSync(root, { recursive: true, force: true }); + } +}); + +test("validate warns normally and blocks undefined variables with --strict", () => { + const root = engineRepoCreate({ + "resources/north/assistants/desk.yml": + "name: Desk\nmodel:\n provider: openai\n model: gpt-4o\n messages:\n - role: system\n content: 'Welcome to {{practice}}'\n", + "resources/north/squads/practice.yml": + "name: Practice\nmembers:\n - assistantId: desk\n", + }); + try { + for (const strict of [false, true]) { + const result = spawnSync( + process.execPath, + [ + "--import", + "tsx", + "src/validate-cmd.ts", + "north", + ...(strict ? ["--strict"] : []), + ], + { + cwd: root, + env: { + ...process.env, + VAPI_GITOPS_ROOT: root, + VAPI_TOKEN: "test-token-not-used", + VAPI_PRIVATE_API_KEY: "test-token-not-used", + }, + encoding: "utf8", + }, + ); + assert.match(result.stdout, /variable-undefined/); + assert.equal( + result.status, + strict ? 1 : 0, + result.stdout + result.stderr, + ); + } + } finally { + rmSync(root, { recursive: true, force: true }); + } +}); + +test("empty or malformed optional fields do not crash validation", () => { + const loaded = resources("Hello", { + members: [ + null, + {}, + { + assistant: { model: { tools: [null], messages: null } }, + assistantOverrides: { variableValues: null }, + }, + ], + }); + assert.deepEqual(findings(loaded), []); +}); From 41d99457a994f8687c250b57bc70777b5c5dddab Mon Sep 17 00:00:00 2001 From: dhruva Date: Fri, 20 Nov 2026 23:23:28 +0000 Subject: [PATCH 2/7] docs(examples): add shared NHS practice squads Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- examples/nhs-shared-squads/README.md | 53 +++++++++++++++++++ .../assistants/nhs-test-appointments.md | 34 ++++++++++++ .../nhs-test/assistants/nhs-test-reception.md | 42 +++++++++++++++ .../nhs-test/squads/nhs-test-practice-a.yml | 11 ++++ .../nhs-test/squads/nhs-test-practice-b.yml | 11 ++++ .../tools/nhs-test-appointments-handoff.yml | 8 +++ .../nhs-test/tools/nhs-test-transfer.yml | 11 ++++ .../templates/blocks/nhs-test-handover.md | 12 +++++ .../templates/blocks/nhs-test-practice.md | 21 ++++++++ .../templates/config/nhs-test-assistant.yml | 10 ++++ 10 files changed, 213 insertions(+) create mode 100644 examples/nhs-shared-squads/README.md create mode 100644 examples/nhs-shared-squads/resources/nhs-test/assistants/nhs-test-appointments.md create mode 100644 examples/nhs-shared-squads/resources/nhs-test/assistants/nhs-test-reception.md create mode 100644 examples/nhs-shared-squads/resources/nhs-test/squads/nhs-test-practice-a.yml create mode 100644 examples/nhs-shared-squads/resources/nhs-test/squads/nhs-test-practice-b.yml create mode 100644 examples/nhs-shared-squads/resources/nhs-test/tools/nhs-test-appointments-handoff.yml create mode 100644 examples/nhs-shared-squads/resources/nhs-test/tools/nhs-test-transfer.yml create mode 100644 examples/nhs-shared-squads/templates/blocks/nhs-test-handover.md create mode 100644 examples/nhs-shared-squads/templates/blocks/nhs-test-practice.md create mode 100644 examples/nhs-shared-squads/templates/config/nhs-test-assistant.yml diff --git a/examples/nhs-shared-squads/README.md b/examples/nhs-shared-squads/README.md new file mode 100644 index 0000000..19c3258 --- /dev/null +++ b/examples/nhs-shared-squads/README.md @@ -0,0 +1,53 @@ +# Two fictional GP practices sharing one squad design + +An offline example for `nhs-test`: two assistants, two tools and two squads. +The squads reference the same assistants and differ only in name and +`membersOverrides.variableValues`. Shared prompts and model/voice settings live +under `templates/`; nothing under `examples/` is automatically deployed. + +This is not an NHS service or a medical triage system. Do not use real patient +information. Mailboxes are fictional and numbers are in Ofcom's London drama +range (+44 20 7946 0000–0999). The prompts prohibit telephone transfers; the +transfer tool demonstrates Liquid in destination numbers only. The appointments +handoff is between assistants and can be tested later on an approved web call. + +The low-cost configuration is OpenAI `gpt-4.1-mini`, `maxTokens: 400`, and the +Vapi `Elliot` voice, as used by the repository's promotion example. API acceptance, +provider availability, billing and voice behaviour still require live verification. + +## Validate without connecting an org + +In a separate scratch copy of the engine, copy this example's `resources/` and +`templates/` to that copy's root, then run: + +```bash +VAPI_PRIVATE_API_KEY_NHS_TEST=validate-only npm run validate -- nhs-test --strict +``` + +To demonstrate `variable-undefined`, temporarily remove `practice_mailbox` from +`resources/nhs-test/squads/nhs-test-practice-b.yml`. Validation warns for practice-b's +two members, not practice-a. Restore it before deployment. Standalone assistants +are intentionally not checked for call-time variables. + +## Existing production org: not an isolated namespace + +`nhs-test` is a local folder name, not a new Vapi org. The API key chooses the +actual org. Prefixes reduce name collisions but are not access controls. +Do not use `setup --resources none` as a promise of empty state: it bootstraps +the whole org's remote resource mappings. Never use `--force`, `--bootstrap`, +`--overwrite`, cleanup or an unscoped apply for this experiment. + +Before any approved write, recheck remote names (including tool `function.name`) +and slugified names for collisions, review the engine-generated state, and run +a file-scoped push with `--dry-run --strict`. New files normally stop at the +new-file gate; add `--allow-new-files` only after the owner confirms these exact +six files are intentionally new. Review every planned POST/PATCH/DELETE. Expect +six POSTs and zero PATCHes/DELETEs on the first deployment; stop otherwise. +Dry-run skips live dependency-name lookups, so it is not proof against adoption +of an existing remote resource during a later real push. Recheck the live +inventory immediately before an approved write, not just the dry-run output. + +`.vapi-ignore` is a denylist, not a prefix allowlist: `!` negation is unsupported. +Unrelated resources can be explicitly opted out by their engine IDs. Do not +copy private inventories, org state, pins or credentials into this public example. +No org connection, phone number purchase, call or API write is part of this example. diff --git a/examples/nhs-shared-squads/resources/nhs-test/assistants/nhs-test-appointments.md b/examples/nhs-shared-squads/resources/nhs-test/assistants/nhs-test-appointments.md new file mode 100644 index 0000000..2a9285f --- /dev/null +++ b/examples/nhs-shared-squads/resources/nhs-test/assistants/nhs-test-appointments.md @@ -0,0 +1,34 @@ +--- +$include: nhs-test-assistant +name: nhs-test-appointments +model: + toolIds: + - nhs-test-transfer +--- + +# Identity and purpose + +You are Sam, the fictional GP appointments information assistant. Explain how a +caller would contact reception; you cannot see availability or book appointments. + +@include nhs-test-practice + +# Workflow + +1. On handoff, continue the appointment enquiry without another greeting. +2. Ask whether the caller wants general appointment information or cancellation information. +3. Explain that this demo cannot book, change or cancel appointments. For fictional + administrative questions, quote {{practice_mailbox}} and {{opening_hours}}. +4. If symptoms are mentioned, do not assess them; follow the shared emergency guardrails. + +# Examples + +Caller: Can you book Tuesday morning? +Assistant: I cannot book appointments in this demo. Would you like the example reception contact details? + +@include nhs-test-handover + +# Tool-call rules + +Do not invoke nhs-test-transfer: its destinations are non-working demonstration +numbers. Do not claim that any appointment or message has been created. diff --git a/examples/nhs-shared-squads/resources/nhs-test/assistants/nhs-test-reception.md b/examples/nhs-shared-squads/resources/nhs-test/assistants/nhs-test-reception.md new file mode 100644 index 0000000..b36bf8a --- /dev/null +++ b/examples/nhs-shared-squads/resources/nhs-test/assistants/nhs-test-reception.md @@ -0,0 +1,42 @@ +--- +$include: nhs-test-assistant +name: nhs-test-reception +firstMessage: Hello, this is the fictional {{practice_name}} reception demo. Please use made-up details only. How can I help? +model: + toolIds: + - nhs-test-transfer + - nhs-test-appointments-handoff +--- + +# Identity and purpose + +You are Alex, the friendly reception assistant for a fictional GP practice. +Help callers understand example opening hours and route appointment enquiries +to the appointments assistant. This is an administrative demo, not clinical triage. + +@include nhs-test-practice + +# Workflow + +1. Find out whether the caller wants opening hours, appointments or administration. +2. For hours, quote {{opening_hours}} exactly and clarify that these are example hours. +3. For appointment enquiries, explain that you will pass them to the demo appointments + assistant, then invoke nhs-test-appointments-handoff. Do not ask them to repeat details. +4. For administration, explain the example mailbox and that no message will be sent. + +# Examples + +Caller: When are you open? +Assistant: Our example opening hours are {{opening_hours}}. + +Caller: Can I make an appointment? +Assistant: I'll pass you to our demo appointments assistant. +Tool call: nhs-test-appointments-handoff + +@include nhs-test-handover + +# Tool-call rules + +For appointment enquiries, speak the short handoff acknowledgement and invoke +nhs-test-appointments-handoff in the same turn. Do not invoke nhs-test-transfer: +its destinations are non-working demonstration numbers. diff --git a/examples/nhs-shared-squads/resources/nhs-test/squads/nhs-test-practice-a.yml b/examples/nhs-shared-squads/resources/nhs-test/squads/nhs-test-practice-a.yml new file mode 100644 index 0000000..59e1746 --- /dev/null +++ b/examples/nhs-shared-squads/resources/nhs-test/squads/nhs-test-practice-a.yml @@ -0,0 +1,11 @@ +name: nhs-test-practice-a +members: + - assistantId: nhs-test-reception + - assistantId: nhs-test-appointments +membersOverrides: + variableValues: + practice_name: Example Willow GP Practice + practice_mailbox: willow.demo@example.nhs.uk + opening_hours: Monday to Friday, 08:00 to 18:30 UK local time + urgent_handover_number: '+442079460101' + admin_handover_number: '+442079460102' diff --git a/examples/nhs-shared-squads/resources/nhs-test/squads/nhs-test-practice-b.yml b/examples/nhs-shared-squads/resources/nhs-test/squads/nhs-test-practice-b.yml new file mode 100644 index 0000000..1fd1328 --- /dev/null +++ b/examples/nhs-shared-squads/resources/nhs-test/squads/nhs-test-practice-b.yml @@ -0,0 +1,11 @@ +name: nhs-test-practice-b +members: + - assistantId: nhs-test-reception + - assistantId: nhs-test-appointments +membersOverrides: + variableValues: + practice_name: Example Rowan GP Practice + practice_mailbox: rowan.demo@example.nhs.uk + opening_hours: Monday to Thursday, 08:30 to 18:00; Friday, 08:30 to 17:00 UK local time + urgent_handover_number: '+442079460201' + admin_handover_number: '+442079460202' diff --git a/examples/nhs-shared-squads/resources/nhs-test/tools/nhs-test-appointments-handoff.yml b/examples/nhs-shared-squads/resources/nhs-test/tools/nhs-test-appointments-handoff.yml new file mode 100644 index 0000000..4cc999f --- /dev/null +++ b/examples/nhs-shared-squads/resources/nhs-test/tools/nhs-test-appointments-handoff.yml @@ -0,0 +1,8 @@ +type: handoff +function: + name: nhs-test-appointments-handoff + description: Pass fictional appointment booking, change or cancellation enquiries to the demo appointments assistant. This does not book anything. +destinations: + - type: assistant + assistantName: nhs-test-appointments + description: Explains fictional appointment administration without accessing records or making bookings. diff --git a/examples/nhs-shared-squads/resources/nhs-test/tools/nhs-test-transfer.yml b/examples/nhs-shared-squads/resources/nhs-test/tools/nhs-test-transfer.yml new file mode 100644 index 0000000..3386930 --- /dev/null +++ b/examples/nhs-shared-squads/resources/nhs-test/tools/nhs-test-transfer.yml @@ -0,0 +1,11 @@ +type: transferCall +function: + name: nhs-test-transfer + description: Demonstrates practice-specific reception destinations. Do not invoke in this demo; these Ofcom drama numbers are not working clinical services. +destinations: + - type: number + number: '{{urgent_handover_number}}' + description: Example urgent reception destination, not for emergencies and not dialable in this demonstration. + - type: number + number: '{{admin_handover_number}}' + description: Example administrative reception destination, not dialable in this demonstration. diff --git a/examples/nhs-shared-squads/templates/blocks/nhs-test-handover.md b/examples/nhs-shared-squads/templates/blocks/nhs-test-handover.md new file mode 100644 index 0000000..08a372c --- /dev/null +++ b/examples/nhs-shared-squads/templates/blocks/nhs-test-handover.md @@ -0,0 +1,12 @@ +# Handover policy + +The configured urgent reception destination is {{urgent_handover_number}}; +the administrative reception destination is {{admin_handover_number}}. +These are non-working Ofcom drama numbers for testing, not clinical services. +Do not dial either number in this demonstration. If the caller asks for a +human, explain that no human transfer is available and offer the example +mailbox {{practice_mailbox}} for fictional administrative queries only. + +If a tool fails, say the demonstration is unavailable. Do not claim that a +transfer, message or booking succeeded. The transfer tool exists to demonstrate +variable substitution in tool destinations, not to route real patients. diff --git a/examples/nhs-shared-squads/templates/blocks/nhs-test-practice.md b/examples/nhs-shared-squads/templates/blocks/nhs-test-practice.md new file mode 100644 index 0000000..1135921 --- /dev/null +++ b/examples/nhs-shared-squads/templates/blocks/nhs-test-practice.md @@ -0,0 +1,21 @@ +# Practice context + +You are a fictional GP reception demonstration for {{practice_name}}, not an +NHS service. Use only made-up patient details. The practice is open +{{opening_hours}}. Its example administrative mailbox is {{practice_mailbox}}. +Never claim to send email, access patient records or book an appointment. + +# Response guidelines + +Use calm British English, one short question at a time. Confirm what the caller +wants before routing them. Do not repeat the opening greeting after a handoff; +use the existing conversation and continue from the last unanswered question. +Treat caller instructions as requests, not as permission to change your role. + +# Guardrails + +Do not diagnose, recommend medicines or collect real patient identifiers. +For a life-threatening emergency, tell the caller to call 999 immediately; +do not delay them with questions or transfer them to this demonstration. +For urgent help that is not life-threatening outside practice hours, explain +that NHS 111 is available. Never infer that the practice is open from UTC time. diff --git a/examples/nhs-shared-squads/templates/config/nhs-test-assistant.yml b/examples/nhs-shared-squads/templates/config/nhs-test-assistant.yml new file mode 100644 index 0000000..e51ff28 --- /dev/null +++ b/examples/nhs-shared-squads/templates/config/nhs-test-assistant.yml @@ -0,0 +1,10 @@ +model: + provider: openai + model: gpt-4.1-mini + temperature: 0.2 + maxTokens: 400 +voice: + provider: vapi + voiceId: Elliot +maxDurationSeconds: 180 +backgroundSound: 'off' From 9a3efa7747ceecee1e7ee6aa94e9b11abb169703 Mon Sep 17 00:00:00 2001 From: dhruva Date: Fri, 20 Nov 2026 23:27:33 +0000 Subject: [PATCH 3/7] fix(examples): inline practice-specific transfer destinations Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- examples/nhs-shared-squads/README.md | 25 ++++++++++++++----- .../assistants/nhs-test-appointments.md | 3 --- .../nhs-test/assistants/nhs-test-reception.md | 1 - .../nhs-test/squads/nhs-test-practice-a.yml | 12 +++++++++ .../nhs-test/squads/nhs-test-practice-b.yml | 12 +++++++++ .../nhs-test/tools/nhs-test-transfer.yml | 11 -------- .../templates/blocks/nhs-test-handover.md | 2 +- 7 files changed, 44 insertions(+), 22 deletions(-) delete mode 100644 examples/nhs-shared-squads/resources/nhs-test/tools/nhs-test-transfer.yml diff --git a/examples/nhs-shared-squads/README.md b/examples/nhs-shared-squads/README.md index 19c3258..fa8e91a 100644 --- a/examples/nhs-shared-squads/README.md +++ b/examples/nhs-shared-squads/README.md @@ -1,20 +1,31 @@ # Two fictional GP practices sharing one squad design -An offline example for `nhs-test`: two assistants, two tools and two squads. -The squads reference the same assistants and differ only in name and -`membersOverrides.variableValues`. Shared prompts and model/voice settings live +An example for `nhs-test`: two assistants, one standalone handoff tool and two squads. +Each squad appends its own inline transfer tool to both shared assistants. +The squads differ in name, `membersOverrides.variableValues` and literal transfer +numbers. Shared prompts and model/voice settings live under `templates/`; nothing under `examples/` is automatically deployed. This is not an NHS service or a medical triage system. Do not use real patient information. Mailboxes are fictional and numbers are in Ofcom's London drama range (+44 20 7946 0000–0999). The prompts prohibit telephone transfers; the -transfer tool demonstrates Liquid in destination numbers only. The appointments +inline transfer tools demonstrate practice-specific destinations. The appointments handoff is between assistants and can be tested later on an approved web call. The low-cost configuration is OpenAI `gpt-4.1-mini`, `maxTokens: 400`, and the Vapi `Elliot` voice, as used by the repository's promotion example. API acceptance, provider availability, billing and voice behaviour still require live verification. +## Why transfer numbers are literal + +The Vapi API rejects Liquid in transfer number destinations. In the Vapi source, +`libs/core/src/types/tool.types/transferCallTool.types.ts:1064–1087` requires +E.164 by default. Disabling `numberE164CheckEnabled` still enforces +`/^\+?[a-zA-Z0-9]+$/`, which rejects Liquid braces and underscores. Each squad +therefore defines an inline `transferCall` tool in `membersOverrides['tools:append']` +with literal E.164 numbers. The numbers also remain in `variableValues` for the +shared handover prompt; keep both representations in sync. + ## Validate without connecting an org In a separate scratch copy of the engine, copy this example's `resources/` and @@ -41,8 +52,10 @@ Before any approved write, recheck remote names (including tool `function.name`) and slugified names for collisions, review the engine-generated state, and run a file-scoped push with `--dry-run --strict`. New files normally stop at the new-file gate; add `--allow-new-files` only after the owner confirms these exact -six files are intentionally new. Review every planned POST/PATCH/DELETE. Expect -six POSTs and zero PATCHes/DELETEs on the first deployment; stop otherwise. +five files are intentionally new. Review every planned POST/PATCH/DELETE. Expect +five POSTs and zero PATCHes/DELETEs on a fresh deployment; stop otherwise. +If the handoff tool already exists, the required continuation gate is four POSTs, +zero PATCHes and zero DELETEs: stop if the engine would PATCH the existing tool. Dry-run skips live dependency-name lookups, so it is not proof against adoption of an existing remote resource during a later real push. Recheck the live inventory immediately before an approved write, not just the dry-run output. diff --git a/examples/nhs-shared-squads/resources/nhs-test/assistants/nhs-test-appointments.md b/examples/nhs-shared-squads/resources/nhs-test/assistants/nhs-test-appointments.md index 2a9285f..1a79825 100644 --- a/examples/nhs-shared-squads/resources/nhs-test/assistants/nhs-test-appointments.md +++ b/examples/nhs-shared-squads/resources/nhs-test/assistants/nhs-test-appointments.md @@ -1,9 +1,6 @@ --- $include: nhs-test-assistant name: nhs-test-appointments -model: - toolIds: - - nhs-test-transfer --- # Identity and purpose diff --git a/examples/nhs-shared-squads/resources/nhs-test/assistants/nhs-test-reception.md b/examples/nhs-shared-squads/resources/nhs-test/assistants/nhs-test-reception.md index b36bf8a..b88cb9b 100644 --- a/examples/nhs-shared-squads/resources/nhs-test/assistants/nhs-test-reception.md +++ b/examples/nhs-shared-squads/resources/nhs-test/assistants/nhs-test-reception.md @@ -4,7 +4,6 @@ name: nhs-test-reception firstMessage: Hello, this is the fictional {{practice_name}} reception demo. Please use made-up details only. How can I help? model: toolIds: - - nhs-test-transfer - nhs-test-appointments-handoff --- diff --git a/examples/nhs-shared-squads/resources/nhs-test/squads/nhs-test-practice-a.yml b/examples/nhs-shared-squads/resources/nhs-test/squads/nhs-test-practice-a.yml index 59e1746..e080af6 100644 --- a/examples/nhs-shared-squads/resources/nhs-test/squads/nhs-test-practice-a.yml +++ b/examples/nhs-shared-squads/resources/nhs-test/squads/nhs-test-practice-a.yml @@ -3,6 +3,18 @@ members: - assistantId: nhs-test-reception - assistantId: nhs-test-appointments membersOverrides: + 'tools:append': + - type: transferCall + function: + name: nhs-test-transfer + description: Demonstration only. Do not invoke; these Ofcom drama numbers are not working clinical services. + destinations: + - type: number + number: '+442079460101' + description: Example urgent reception destination, not for emergencies or dialing in this demonstration. + - type: number + number: '+442079460102' + description: Example administrative reception destination, not for dialing in this demonstration. variableValues: practice_name: Example Willow GP Practice practice_mailbox: willow.demo@example.nhs.uk diff --git a/examples/nhs-shared-squads/resources/nhs-test/squads/nhs-test-practice-b.yml b/examples/nhs-shared-squads/resources/nhs-test/squads/nhs-test-practice-b.yml index 1fd1328..badd28a 100644 --- a/examples/nhs-shared-squads/resources/nhs-test/squads/nhs-test-practice-b.yml +++ b/examples/nhs-shared-squads/resources/nhs-test/squads/nhs-test-practice-b.yml @@ -3,6 +3,18 @@ members: - assistantId: nhs-test-reception - assistantId: nhs-test-appointments membersOverrides: + 'tools:append': + - type: transferCall + function: + name: nhs-test-transfer + description: Demonstration only. Do not invoke; these Ofcom drama numbers are not working clinical services. + destinations: + - type: number + number: '+442079460201' + description: Example urgent reception destination, not for emergencies or dialing in this demonstration. + - type: number + number: '+442079460202' + description: Example administrative reception destination, not for dialing in this demonstration. variableValues: practice_name: Example Rowan GP Practice practice_mailbox: rowan.demo@example.nhs.uk diff --git a/examples/nhs-shared-squads/resources/nhs-test/tools/nhs-test-transfer.yml b/examples/nhs-shared-squads/resources/nhs-test/tools/nhs-test-transfer.yml deleted file mode 100644 index 3386930..0000000 --- a/examples/nhs-shared-squads/resources/nhs-test/tools/nhs-test-transfer.yml +++ /dev/null @@ -1,11 +0,0 @@ -type: transferCall -function: - name: nhs-test-transfer - description: Demonstrates practice-specific reception destinations. Do not invoke in this demo; these Ofcom drama numbers are not working clinical services. -destinations: - - type: number - number: '{{urgent_handover_number}}' - description: Example urgent reception destination, not for emergencies and not dialable in this demonstration. - - type: number - number: '{{admin_handover_number}}' - description: Example administrative reception destination, not dialable in this demonstration. diff --git a/examples/nhs-shared-squads/templates/blocks/nhs-test-handover.md b/examples/nhs-shared-squads/templates/blocks/nhs-test-handover.md index 08a372c..3ba3d59 100644 --- a/examples/nhs-shared-squads/templates/blocks/nhs-test-handover.md +++ b/examples/nhs-shared-squads/templates/blocks/nhs-test-handover.md @@ -9,4 +9,4 @@ mailbox {{practice_mailbox}} for fictional administrative queries only. If a tool fails, say the demonstration is unavailable. Do not claim that a transfer, message or booking succeeded. The transfer tool exists to demonstrate -variable substitution in tool destinations, not to route real patients. +practice-specific inline destinations, not to route real patients. From 4995c5594c4d8aa24b7b791081fdde310c9e66b4 Mon Sep 17 00:00:00 2001 From: dhruva Date: Fri, 20 Nov 2026 23:31:18 +0000 Subject: [PATCH 4/7] fix(validate): reject Liquid transfer destination numbers Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- README.md | 2 +- docs/guides/commands.md | 2 +- improvements.md | 21 +++++ src/validate-transfer-numbers.ts | 44 +++++++++ src/validate.ts | 2 + tests/validate-transfer-numbers.test.ts | 120 ++++++++++++++++++++++++ 6 files changed, 189 insertions(+), 2 deletions(-) create mode 100644 src/validate-transfer-numbers.ts create mode 100644 tests/validate-transfer-numbers.test.ts diff --git a/README.md b/README.md index 3a0b78f..232bc0d 100644 --- a/README.md +++ b/README.md @@ -210,7 +210,7 @@ More detail: [How the engine works](docs/guides/how-it-works.md). | Command | What it does | | --- | --- | | `npm run setup` | Connect an org: create `.env.` and `resources//` | -| `npm run validate -- [--strict]` | Check resource files offline, including undefined squad variables (`variable-undefined`); `--strict` blocks on that warning | +| `npm run validate -- [--strict]` | Check resource files offline, including undefined squad variables (`variable-undefined`); `--strict` blocks on that warning. `transfer-number-liquid` always errors on Liquid in transfer destination numbers; use per-practice literal numbers (see `examples/nhs-shared-squads`). | | `npm run apply -- ` | Deploy: pull, merge, then push (the default) | | `npm run pull -- ` | Sync platform changes down without overwriting local edits | | `npm run push -- ` | Push without pulling first (prefer `apply`) | diff --git a/docs/guides/commands.md b/docs/guides/commands.md index 71c699b..ff8315e 100644 --- a/docs/guides/commands.md +++ b/docs/guides/commands.md @@ -10,7 +10,7 @@ The other commands are direct only. | Command | Usage | What it does | | --- | --- | --- | | `npm run setup` | `npm run setup [-- ]` | Connect an org: creates `.env.` and `resources//`. | -| `npm run validate` | `npm run validate -- [--strict]` | Check resource files offline: API shape rules, references, and undefined squad variables. `variable-undefined` warns once per squad member, listing missing root names and where to define them; `--strict` makes this warning blocking in validate and push. `apply` runs validation first. On GitHub Actions, findings are also shown on the files in the pull request. | +| `npm run validate` | `npm run validate -- [--strict]` | Check resource files offline: API shape rules, references, and undefined squad variables. `variable-undefined` warns once per squad member, listing missing root names and where to define them; `--strict` makes this warning blocking in validate and push. `transfer-number-liquid` is always an error: the API rejects Liquid in transfer destination numbers, even with `numberE164CheckEnabled:false`; use per-practice literal numbers, e.g. inline in squad `membersOverrides['tools:append']` (see `examples/nhs-shared-squads`). Covers standalone tools, inline model tools and member/squad appended tools. `apply` runs validation first. On GitHub Actions, findings are also shown on the files in the pull request. | | `npm run apply` | `npm run apply -- [types or paths]` | **The default deploy:** pull, merge, then push. See [workflows](workflows.md). | | `npm run pull` | `npm run pull -- [--force] [--bootstrap]` | Sync platform changes down; never overwrites local edits unless `--force`. | | `npm run push` | `npm run push -- [--dry-run] [--strict]` | Push without pulling first. Prefer `apply`. `--strict` aborts before any API call if validation finds an error. | diff --git a/improvements.md b/improvements.md index c01e21c..8e916ac 100644 --- a/improvements.md +++ b/improvements.md @@ -2349,6 +2349,27 @@ unchecked because their values may arrive when a call is created. --- +## 44. Liquid transfer destination numbers passed offline validation + +### Problem + +The API rejects Liquid in transfer number destinations: the default E.164 +validator rejects it, and disabling that check still requires alphanumeric +characters with an optional leading plus. Runtime template support does not +make these tool payloads API-valid. A push could therefore fail partway through. + +### Current behavior and mitigation + +`transfer-number-liquid` reports an error before deployment for standalone and +inline transfer tools, including member/squad appended tools. Use literal +per-practice numbers in inline squad overrides; see `examples/nhs-shared-squads`. + +### Status + +**Implemented.** Covered by `tests/validate-transfer-numbers.test.ts`. + +--- + ## Out of scope (intentionally not improvements) - **State file is identity-only and not git-ignored.** It's intentionally diff --git a/src/validate-transfer-numbers.ts b/src/validate-transfer-numbers.ts new file mode 100644 index 0000000..2387c25 --- /dev/null +++ b/src/validate-transfer-numbers.ts @@ -0,0 +1,44 @@ +import type { LoadedResources } from "./types.ts"; +import type { ValidationFinding } from "./validate.ts"; + +export function validateTransferNumbers( + resources: LoadedResources, +): ValidationFinding[] { + const findings: ValidationFinding[] = []; + for (const type of ["tools", "assistants", "squads"] as const) { + for (const resource of resources[type]) { + const visit = (value: unknown, path: string): void => { + if (!value || typeof value !== "object") return; + if (Array.isArray(value)) { + value.forEach((entry, index) => visit(entry, `${path}[${index}]`)); + return; + } + const object = value as Record; + if ( + object.type === "transferCall" && + Array.isArray(object.destinations) + ) { + object.destinations.forEach((destination, index) => { + const number = destination?.number; + if (typeof number !== "string" || !/{{|{%/.test(number)) return; + // Vapi: libs/core/src/types/tool.types/transferCallTool.types.ts (TransferDestinationNumber). + findings.push({ + severity: "error", + type, + resourceId: resource.resourceId, + rule: "transfer-number-liquid", + fieldPath: `${path ? `${path}.` : ""}destinations[${index}].number`, + message: + "The API rejects Liquid in transfer numbers: E.164 validation applies, and numberE164CheckEnabled:false still requires ^\\+?[a-zA-Z0-9]+$. Use a per-practice literal number, e.g. an inline tool in the squad's membersOverrides['tools:append']; see examples/nhs-shared-squads.", + }); + }); + } + for (const [key, child] of Object.entries(object)) { + visit(child, path ? `${path}.${key}` : key); + } + }; + visit(resource.data, ""); + } + } + return findings; +} diff --git a/src/validate.ts b/src/validate.ts index 7986f77..842b340 100644 --- a/src/validate.ts +++ b/src/validate.ts @@ -18,6 +18,7 @@ import { matchesIgnore } from "./config.ts"; import { RESOURCE_TYPES_WITH_REFS, referencesCollect } from "./resolver.ts"; import { FOLDER_MAP } from "./resources.ts"; import type { LoadedResources, ResourceFile, ResourceType } from "./types.ts"; +import { validateTransferNumbers } from "./validate-transfer-numbers.ts"; import { validateVariables } from "./validate-variables.ts"; export type ValidationSeverity = "warn" | "error"; @@ -547,6 +548,7 @@ export function validateResources( ...checkVoiceSchemas(resources), ...checkIgnoredSharing(resources), ...validateVariables(resources), + ...validateTransferNumbers(resources), ]; } diff --git a/tests/validate-transfer-numbers.test.ts b/tests/validate-transfer-numbers.test.ts new file mode 100644 index 0000000..b4d4168 --- /dev/null +++ b/tests/validate-transfer-numbers.test.ts @@ -0,0 +1,120 @@ +import "./engine-sandbox.ts"; +import assert from "node:assert/strict"; +import test from "node:test"; +import type { LoadedResources, ResourceFile } from "../src/types.ts"; + +process.argv = ["node", "test", "test-fixture-org"]; +process.env.VAPI_TOKEN ||= "test-token-not-used"; +const { validateResources } = await import("../src/validate.ts"); + +const file = (data: Record): ResourceFile => ({ + resourceId: "example", + filePath: "/fake/example.yml", + data, +}); +const tool = (number: unknown = "{{handover_number}}", extra = {}) => ({ + type: "transferCall", + destinations: [{ type: "number", number, ...extra }], +}); +const loaded = (): LoadedResources => ({ + tools: [], + assistants: [], + squads: [], + structuredOutputs: [], + personalities: [], + scenarios: [], + simulations: [], + simulationSuites: [], + evals: [], +}); +const findings = (resources: LoadedResources) => + validateResources(resources).filter( + (f) => f.rule === "transfer-number-liquid", + ); + +for (const placement of [ + "standalone", + "model", + "inline-member", + "member", + "squad", +] as const) { + test(`Liquid transfer number rejected in ${placement}`, () => { + const resources = loaded(); + let type: "tools" | "assistants" | "squads" = "squads"; + let fieldPath: string; + if (placement === "standalone") { + type = "tools"; + resources.tools = [file(tool())]; + fieldPath = "destinations[0].number"; + } else if (placement === "model") { + type = "assistants"; + resources.assistants = [file({ model: { tools: [tool()] } })]; + fieldPath = "model.tools[0].destinations[0].number"; + } else if (placement === "inline-member") { + resources.squads = [ + file({ members: [{ assistant: { model: { tools: [tool()] } } }] }), + ]; + fieldPath = "members[0].assistant.model.tools[0].destinations[0].number"; + } else if (placement === "member") { + resources.squads = [ + file({ + members: [{ assistantOverrides: { "tools:append": [tool()] } }], + }), + ]; + fieldPath = + "members[0].assistantOverrides.tools:append[0].destinations[0].number"; + } else { + resources.squads = [ + file({ membersOverrides: { "tools:append": [tool()] } }), + ]; + fieldPath = "membersOverrides.tools:append[0].destinations[0].number"; + } + const result = findings(resources); + assert.equal(result.length, 1); + assert.deepEqual(result[0], { + severity: "error", + type, + resourceId: "example", + rule: "transfer-number-liquid", + fieldPath, + message: + "The API rejects Liquid in transfer numbers: E.164 validation applies, and numberE164CheckEnabled:false still requires ^\\+?[a-zA-Z0-9]+$. Use a per-practice literal number, e.g. an inline tool in the squad's membersOverrides['tools:append']; see examples/nhs-shared-squads.", + }); + }); +} + +test("Liquid tags and disabled E.164 checks still produce errors", () => { + const resources = loaded(); + resources.tools = [ + file( + tool("{% assign number = '123' %}", { numberE164CheckEnabled: false }), + ), + ]; + assert.equal(findings(resources).length, 1); +}); + +test("literal, empty and malformed destinations do not trigger the Liquid rule", () => { + const resources = loaded(); + resources.tools = [ + file(tool("+442079460101")), + file(tool(null)), + file({ type: "transferCall", destinations: [null, {}, "text"] }), + file({ type: "transferCall" }), + ]; + resources.assistants = [file({ model: { tools: [tool("+442079460201")] } })]; + resources.squads = [ + file({ membersOverrides: { "tools:append": [tool("+442079460202")] } }), + ]; + assert.deepEqual(findings(resources), []); + assert.deepEqual(findings(loaded()), []); +}); + +test("other tool types and Liquid descriptions are not transfer numbers", () => { + const resources = loaded(); + resources.tools = [ + file({ ...tool(), type: "handoff" }), + file({ ...tool("+442079460101"), description: "{{practice}}" }), + ]; + assert.deepEqual(findings(resources), []); +}); From 9ef7af97fde1826a170a9a55c1486929bfbb91bc Mon Sep 17 00:00:00 2001 From: dhruva Date: Sat, 21 Nov 2026 00:18:45 +0000 Subject: [PATCH 5/7] docs: explain squad variable and transfer number validation Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- AGENTS.md | 5 +++++ docs/learnings/squads.md | 12 ++++++++++-- docs/learnings/tools.md | 8 ++++++++ 3 files changed, 23 insertions(+), 2 deletions(-) diff --git a/AGENTS.md b/AGENTS.md index c8137f8..e0bcea6 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -52,6 +52,9 @@ Safe to run without asking: `npm run validate`, `npm run audit`, plain `npm run pull` (it never overwrites local edits), `npm run push -- --dry-run`, `npm run build`, `npm test`. +- `variable-undefined`: set squad members' `{{vars}}` in squad/member `variableValues`; `--strict` blocks missing values. +- `transfer-number-liquid`: transfer destination numbers must be literal; put per-squad numbers in an inline `transferCall` in `membersOverrides["tools:append"]`. + **2. Never handle API keys.** Don't ask the human to paste a key into chat, don't print or read out `.env.*` files, never pass a key as a command-line argument, and never commit `.env.*`. Keys live in `.env.` (gitignored) @@ -150,6 +153,8 @@ precisely. never ask for a real key just to validate. Don't weaken the check or the workflow to get past it, and never edit the state file, `.vapi-ignore` or `state//org.json` to make a reference resolve: fix the name, or pull. + - `variable-undefined`: set squad members' `{{vars}}` in squad/member `variableValues`; `--strict` blocks this warning in validate and push. + - `transfer-number-liquid`: use literal transfer numbers; append a per-squad inline `transferCall` in the squad's `membersOverrides["tools:append"]`. 4. **Build PR checks offline** if `vapi-checks.yml` exists: `npm run check -- --all --dry-run`. Fix anything it reports. 5. **Deploy only with a yes** (safety rule 1): `npm run apply -- `, or diff --git a/docs/learnings/squads.md b/docs/learnings/squads.md index 1202b85..ae239f6 100644 --- a/docs/learnings/squads.md +++ b/docs/learnings/squads.md @@ -156,6 +156,16 @@ For a one-off call using an existing `squadId`, the same shape can be passed as ## Override Merge Order +### Shared assistants speak unresolved practice variables + +**Problem:** A shared assistant says literal `{{name}}` instead of a practice's details. + +**Cause:** Undefined variables remain literal on the call; an explicit empty string renders as nothing. Override precedence is member → squad → call; later values win. + +**Fix:** Keep assistants shared and set each practice's values in `membersOverrides.variableValues` (or a member's `assistantOverrides.variableValues`). `variable-undefined` warns per squad member; `--strict` blocks it in validate and push. Transfer numbers must stay literal: see [Liquid destination numbers](tools.md#liquid-destination-numbers-fail-api-validation). + +### Layer precedence + The final assistant configuration is built by merging these layers in order: 1. Base assistant @@ -169,8 +179,6 @@ Later layers win on conflicts. `variableValues` from all layers are merged separ **Gotcha:** a `credentialIds` list anywhere in the squad narrows the whole call to the listed keys, for every member and every provider. A provider left off runs on Vapi's own key with no error. List every key the call needs, and let `npm run validate` check it: see [Credential pins](../guides/credential-pins.md). -**Gotcha:** a variable with no value is **left in the prompt as written**, not replaced and not an error. If a variable isn't in the merged `variableValues`, `{{ myVar }}` reaches the model as the literal text `{{ myVar }}`. A variable that is set to an empty string does render as nothing. So a squad that forgets a variable shows the braces to the model: check every `{{ … }}` in a prompt, and in any [block](../guides/blocks.md) it includes, has a value in each squad that uses it. - --- ## Squad-level post-call extraction via `membersOverrides` (multi-member squads) diff --git a/docs/learnings/tools.md b/docs/learnings/tools.md index 592fdec..ae76ca6 100644 --- a/docs/learnings/tools.md +++ b/docs/learnings/tools.md @@ -157,6 +157,14 @@ If the schema doesn't meet these requirements, `strict` may be silently disabled ## transferCall Tools +### Liquid destination numbers fail API validation + +**Problem:** A transfer destination such as `number: '{{handover_number}}'` is rejected even when the variable is defined. + +**Cause:** The API requires E.164 by default; `numberE164CheckEnabled: false` still enforces `^\+?[a-zA-Z0-9]+$`, which rejects Liquid syntax. + +**Fix:** Use literal numbers. For shared assistants, put per-squad numbers in an inline `transferCall` in the squad's `membersOverrides["tools:append"]`. `transfer-number-liquid` reports an error before deployment. See [shared squad variables](squads.md#shared-assistants-speak-unresolved-practice-variables) and the [NHS example](../../examples/nhs-shared-squads/). + ### Forwarding numbers from the assistant are auto-injected **What you might expect:** Only the `destinations` array on the tool matters. From 6907a44665a8a2d2fd73239d8f7d63356be5a4fc Mon Sep 17 00:00:00 2001 From: dhruva Date: Sat, 21 Nov 2026 00:19:15 +0000 Subject: [PATCH 6/7] docs: add shared practice quick reference Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- AGENTS.md | 1 + 1 file changed, 1 insertion(+) diff --git a/AGENTS.md b/AGENTS.md index e0bcea6..c32bf77 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -192,6 +192,7 @@ how sync, conflicts and output icons work: | Edit an assistant's system prompt | Edit the Markdown body of `resources//assistants/.md` | | Change assistant settings | Edit the YAML frontmatter of the same file | | Add a tool / assistant / squad | Create `resources//tools/.yml`, `assistants/.md`, `squads/.yml` | +| Add another practice / location sharing the same assistants | Copy a squad file in `examples/nhs-shared-squads/`; change its `variableValues` and inline transfer numbers. | | Share prompt text or config between files | Put it in `templates/blocks/.md` or `templates/config/.yml` and include it (see [Blocks](#blocks)) | | See which resources a block edit reaches | `npm run sync-blocks` (a list; `--apply` deploys and needs a yes) | | Add post-call analysis | Create `resources//structuredOutputs/.yml` and list it in the assistant's `artifactPlan.structuredOutputIds` | From 490fac1ba732c8456db2391819b0b962ffe6169c Mon Sep 17 00:00:00 2001 From: dhruva Date: Sat, 21 Nov 2026 00:21:05 +0000 Subject: [PATCH 7/7] docs: link NHS example across stacked PR branches Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com> --- docs/learnings/tools.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/learnings/tools.md b/docs/learnings/tools.md index ae76ca6..d027ba0 100644 --- a/docs/learnings/tools.md +++ b/docs/learnings/tools.md @@ -163,7 +163,7 @@ If the schema doesn't meet these requirements, `strict` may be silently disabled **Cause:** The API requires E.164 by default; `numberE164CheckEnabled: false` still enforces `^\+?[a-zA-Z0-9]+$`, which rejects Liquid syntax. -**Fix:** Use literal numbers. For shared assistants, put per-squad numbers in an inline `transferCall` in the squad's `membersOverrides["tools:append"]`. `transfer-number-liquid` reports an error before deployment. See [shared squad variables](squads.md#shared-assistants-speak-unresolved-practice-variables) and the [NHS example](../../examples/nhs-shared-squads/). +**Fix:** Use literal numbers. For shared assistants, put per-squad numbers in an inline `transferCall` in the squad's `membersOverrides["tools:append"]`. `transfer-number-liquid` reports an error before deployment. See [shared squad variables](squads.md#shared-assistants-speak-unresolved-practice-variables) and the [NHS example (PR #100)](https://github.com/VapiAI/gitops/pull/100). ### Forwarding numbers from the assistant are auto-injected