Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -258,6 +258,7 @@ default-exports the object (safety rule 5).
| Simulation | `personalityId`, `scenarioId` | Personality and scenario files | `scenarioId: books-cleaning` |
| Suite | `simulationIds[]` | Simulation files | `- books-cleaning-calm` |
| Any server block | `credentialId` | A credential **name** in the org | `credentialId: my-api-credential` |
| Assistant, or squad overrides | `credentialIds[]` | Credential **names**. The list is every key the call may use, not one provider's: [credential pins](docs/guides/credential-pins.md) | `- azure-south` |

The engine resolves IDs and credential names to each org's UUIDs on push.

Expand Down
1 change: 1 addition & 0 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -247,6 +247,7 @@ when run without arguments. Full flags and examples:
| [Writing system prompts](docs/guides/writing-prompts.md) | Structure a voice agent's prompt |
| [PR checks](docs/guides/pr-checks.md) | Test every pull request with simulations |
| [Promotion](docs/guides/promotion.md) | Move resources from dev to staging to production |
| [Credential pins](docs/guides/credential-pins.md) | Pin a call to specific keys with `credentialIds`, and mark an org customer-keys-only |
| [How the engine works](docs/guides/how-it-works.md) | Understand sync, references, credentials and state |
| [Configuration](docs/guides/configuration.md) | Environment variables and secrets |
| [Troubleshooting](docs/guides/troubleshooting.md) | Fix common errors |
Expand Down
9 changes: 9 additions & 0 deletions docs/guides/commands.md
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,7 @@ tell it. Nothing changes inside the files.
| --- | --- |
| `.vapi-state.<org>.json` | `state/<org>/state.json` |
| `resources/<org>/.vapi-org.json` | `state/<org>/org.json` |
| `resources/<org>/.vapi-credential-providers.json` | `state/<org>/credential-providers.json` |
| `.vapi-state-hash/<org>/` | `state/<org>/local/baselines/` |
| `.vapi-state.<org>.snapshots/` | `state/<org>/local/snapshots/` |

Expand Down Expand Up @@ -345,6 +346,13 @@ git add state/commit-step.json
baselines yet, the first `pull` after upgrading keeps every local file, and
an edit made in the dashboard comes down on the next `pull`. Nothing is
lost.
- **`credential-pin-unchecked`** from `validate`, once for each org that has a
`credentialIds` list, until that org's credential providers record is
committed. It is a warning and changes no exit code. Run
`npm run pull -- <org> --bootstrap` and commit the file it writes:
`state/<org>/credential-providers.json`, or
`resources/<org>/.vapi-credential-providers.json` while the org's state is
in the older layout. See [credential pins](credential-pins.md).

After `npm run migrate`:

Expand Down Expand Up @@ -373,6 +381,7 @@ for dir in state/*/; do
org=$(basename "$dir")
if [ -f "state/$org/state.json" ]; then mv "state/$org/state.json" ".vapi-state.$org.json"; fi
if [ -f "state/$org/org.json" ]; then mv "state/$org/org.json" "resources/$org/.vapi-org.json"; fi
if [ -f "state/$org/credential-providers.json" ]; then mv "state/$org/credential-providers.json" "resources/$org/.vapi-credential-providers.json"; fi
# This machine's baselines and snapshots. They are not in git, so everyone
# who has pulled or pushed since the move runs these two on their machine.
if [ -d "state/$org/local/baselines" ]; then mkdir -p .vapi-state-hash && mv "state/$org/local/baselines" ".vapi-state-hash/$org"; fi
Expand Down
2 changes: 2 additions & 0 deletions docs/guides/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -49,6 +49,7 @@ Set these in GitHub under **Settings → Secrets and variables → Actions**.
| --- | --- |
| `promotion.yml` | Promotion pipelines between orgs. Start from `promotion.example.yml`. |
| `vapi-checks.yml` | PR checks. Start from `vapi-checks.example.yml`. |
| `resources/<org>/.vapi-policy.yml` | Rules the org must hold, written by hand. `customerKeysOnly: true` makes a `credentialIds` list that hides one of the org's keys an error instead of a warning. See [Credential pins](credential-pins.md). |
| `resources/<org>/.vapi-ignore` | Platform resources this repo shouldn't manage, as glob patterns. See `resources/.vapi-ignore.example` and [YAML conventions](../learnings/yaml-conventions.md). |

These are hand-written. The files the engine writes live apart from them, in
Expand All @@ -58,6 +59,7 @@ These are hand-written. The files the engine writes live apart from them, in
| --- | --- |
| `state/<org>/state.json` | Which platform resource (UUID) each file under `resources/<org>/` is. |
| `state/<org>/org.json` | The Vapi org ID the folder belongs to. Written by the first command that can see the key's org, then checked by every command before any request: a key for another org stops the command. If a folder genuinely moves to another org, delete the file in a reviewed commit. |
| `state/<org>/credential-providers.json` | The provider of each saved credential, by name. Written by every pull and committed with the state file. It holds no secret. The credential-pin check reads it. While an org's state is in the older layout it is `resources/<org>/.vapi-credential-providers.json`. |

## Other environment variables

Expand Down
156 changes: 156 additions & 0 deletions docs/guides/credential-pins.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,156 @@
# Credential pins: `credentialIds` lists

An assistant, a squad member's `assistantOverrides` and a squad's
`membersOverrides` can each carry a `credentialIds` list. It pins a call to
specific keys, for example one site's Azure OpenAI key in an org that holds a
key per site. This page explains what the list does on a call and what
`npm run validate` checks about it.

## What a list does

A `credentialIds` list is not a choice for one provider. It is the full set
of keys the call may use:

- As soon as any assistant on the call has a list, the call can use only the
listed keys. Every squad member's list is pooled into one list for the
whole call, so a list on one member narrows the other members too.
- A provider whose key is not on the list does not fail. The model, voice or
transcriber runs on Vapi's own key, a tool or server request goes out
without its key, and the recording does not go to your storage bucket.
- The phone number's own credential stays usable whatever the list says.

So a list has to name every key the call needs: the model key, the voice
key, the transcriber key, each key a tool or server names in `credentialId`,
the storage bucket key and the Langfuse key if the org has them.

Lists combine like this:

| Where the list is | What happens |
| --- | --- |
| On the assistant | Kept. Lists from overrides are added to it. |
| In a member's `assistantOverrides` | Used, unless the squad's `membersOverrides` also has a list, which replaces it. |
| In the squad's `membersOverrides` | Applies to every member. |

One list per squad, in `membersOverrides`, is the simplest layout: shared
assistants carry no list, and each squad names its own keys.

```yaml
# resources/<org>/squads/south-front-desk.yml
name: South Front Desk
members:
- assistantId: triage
- assistantId: pharmacy
membersOverrides:
credentialIds:
- azure-south # this site's Azure OpenAI key
- elevenlabs # voice
- deepgram # transcriber
- crm-webhook # the key the lookup tool names
- recordings # storage bucket
```

## What validate checks

`npm run validate -- <org>` (and `apply`, which runs it first) reports a list
that leaves out a key **the org has saved** for something the call uses:

| The call uses | Reported when |
| --- | --- |
| A model, voice or transcriber, or a fallback voice or transcriber | The org has a key for that provider and the list has none. An OpenAI model can run on an `openai` key or an `azure-openai` key, so either on the list is enough. Every other provider takes a key saved under its own name. |
| A tool, server or model that names a key in `credentialId` | That key is not on the list. |
| Storage | The org has a storage bucket key and the list has none. |
| Langfuse | The org has a Langfuse key and the list leaves it out. |

It does not report a provider the org has no key of its own for. Running
that provider on Vapi's key is a normal setup, with or without a list.

A different key for the same provider counts as a key for it. A squad pinned
to one site's Azure OpenAI key is not told about the other sites' keys.

For a squad, the check looks at the call as a whole: the pooled list, and
what every member uses after its overrides are applied. An assistant with a
list of its own is also checked on its own, since it can be called outside a
squad.

The finding is `credential-pin-hides-key`:

```text
⚠️ [credential-pin-hides-key] squads/south-front-desk (credentialIds): the voice of member "triage" and the voice of member "pharmacy" use 11labs, and this org has "elevenlabs" saved for it, but the call's credentialIds list ("azure-south", "deepgram") has no key for 11labs. A call that has a credentialIds list can use only the keys on it, so 11labs would run on Vapi's own key, not yours. Add the key to the list
```

### Limits

- The check reads files. A squad member that is not a file in the org's
folder (a dashboard-owned assistant) adds nothing to it.
- It knows each key's provider, not which models or regions the key covers.
A list with an Azure OpenAI key that does not serve the assistant's model
passes.
- A tool or server with no `credentialId` is not checked.
- Lists added at call time (`assistantOverrides` on a call or a phone
number) are outside the repo and are not seen.

## Orgs that must only use their own keys

By default the finding is a warning. For an org whose calls must never run
on Vapi's keys, add a policy file and commit it:

```yaml
# resources/<org>/.vapi-policy.yml
customerKeysOnly: true
```

In that org the finding is an error: `validate` fails, `apply` stops before
it pulls or pushes, and the **Validate resources** check fails on the pull
request. `npm run push` on its own only blocks on errors with `--strict`,
like every other validation error.

The policy belongs to one org. Promotion does not copy it, so set it in each
org that needs it. An unknown key in the file stops `validate`, so a typo
can't switch the rule off.

The policy makes a hidden key an error. It does not require the org to have
a key for every provider it uses.

## How the check knows what each key is for

The state file maps a credential's name to its UUID and nothing else. Pull
also records each key's provider in
`state/<org>/credential-providers.json`:

```json
{
"azure-south": { "provider": "azure-openai" },
"elevenlabs": { "provider": "11labs" },
"recordings": { "bucket": true, "provider": "azure" }
}
```

Commit it with the state file. It holds no secret. Every pull rewrites it,
including `npm run pull -- <org> --bootstrap`, which changes no resource
file. If the file is missing or older than the state file, the check reports
`credential-pin-unchecked` and asks for that pull.

In a repository whose state is still in the layout from before `state/`, the
file is `resources/<org>/.vapi-credential-providers.json` instead, which the
commit step that repository already has stages with the folder. `npm run
migrate` moves it with the rest: see
[Upgrading from an older version](commands.md#upgrading-from-an-older-version).

Give every key a name in Vapi before the first pull. Two keys with no name
and the same provider get the same name here, and only one is kept. Pull
warns when that happens.

## Promotion

A credential with the binding policy `omit` is dropped from a promoted file.
For a single `credentialId` that is the point. For an entry in a
`credentialIds` list it narrows what calls in the destination may use, so
the plan says so:

```text
release: dev → prod
update squads/south-front-desk.yml
⚠️ squads/south-front-desk.yml: credentialIds entry "elevenlabs" is left out in prod (binding policy "omit"). Calls there can use only the keys still on the list, so that provider would run on Vapi's own key.
```

See [Promotion](promotion.md) for bindings.
5 changes: 5 additions & 0 deletions docs/guides/how-it-works.md
Original file line number Diff line number Diff line change
Expand Up @@ -160,6 +160,11 @@ Credentials are managed automatically through the state file. No secrets in reso
2. Resource files use human-readable credential names
3. **Push** resolves names back to UUIDs before sending to the API

Pull also records each credential's provider in
`state/<org>/credential-providers.json`, so `validate` can check a
`credentialIds` list against the keys the org has: see
[Credential pins](credential-pins.md).

Setup and pull also refresh a marked block in the gitignored `.env.<org>` file:

```dotenv
Expand Down
3 changes: 3 additions & 0 deletions docs/guides/promotion.md
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,9 @@ left alone.
5. Choose `bind` or `omit` for credentials and phone numbers in each target
org. Pull generates stable aliases in `.env.<org>` from resources that
already exist there; promotion never copies secrets or provisions numbers.
An `omit` that drops an entry from a `credentialIds` list is flagged in the
plan, because it narrows which keys calls in the target may use: see
[Credential pins](credential-pins.md#promotion).
6. Run the read-only plan for each transition before the first apply and
confirm every create, update, and delete is expected.

Expand Down
4 changes: 4 additions & 0 deletions docs/learnings/squads.md
Original file line number Diff line number Diff line change
Expand Up @@ -165,6 +165,10 @@ The final assistant configuration is built by merging these layers in order:

Later layers win on conflicts. `variableValues` from all layers are merged separately.

`credentialIds` follows the same order with one difference: a list on the base assistant is kept and the override's list is added to it, while `squad.membersOverrides.credentialIds` replaces `member.assistantOverrides.credentialIds`.

**Gotcha:** a `credentialIds` list anywhere in the squad narrows the whole call to the listed keys, for every member and every provider. A provider left off runs on Vapi's own key with no error. List every key the call needs, and let `npm run validate` check it: see [Credential pins](../guides/credential-pins.md).

**Gotcha:** Liquid template substitution replaces undefined variables with **empty strings**, not errors. If a variable isn't in the merged `variableValues`, any `{{ myVar }}` reference silently becomes `""`.

---
Expand Down
59 changes: 59 additions & 0 deletions improvements.md
Original file line number Diff line number Diff line change
Expand Up @@ -110,6 +110,7 @@ you which stack PR closes the row.**
| 37 | Resource validation ran only at deploy time, after merge | A config `apply` refuses could merge and block deploys and promotion | #32 | RESOLVED 2026-10-03 (#76) |
| 38 | Promotion can't carry a simulation that uses a stock personality by UUID | Gated orgs, and any promoted tests, need local personality files | None | Open |
| 39 | Engine state looked like temp files and sat in three places | Deleting the state file makes the next push duplicate every resource | None | RESOLVED 2026-11-17 (#97) |
| 40 | Nothing checked a `credentialIds` list against the keys an org has saved | A list that hides the org's own key runs the call on Vapi's key, silently | None | RESOLVED 2026-11-17 (#96) |

**Active backlog after cleanup:** `#2`, `#6`, `#8`, `#12`, `#20`, `#24–#26`, `#31`, and the open remainder of `#27` (wiring the listing-completeness verdict into push/delete/audit, and moving `cleanup.ts` onto the shared pager). Resolved entries stay in this file as historical incident notes per the maintenance directive; stale superseded backlog rows are not duplicated.

Expand Down Expand Up @@ -2203,6 +2204,64 @@ run's hand-off to the workflow, and an older workflow reads it there.

---

## 40. Nothing checked a `credentialIds` list against the keys an org has saved

**[RESOLVED 2026-11-17] (#96)**

**Discovered:** 2026-11-17, while planning one squad per site in an org that
holds an Azure OpenAI key per site.

### Problem

A `credentialIds` list is the full set of keys a call may use, and every
squad member's list is pooled into one. A provider left off the list does
not fail the call: it runs on Vapi's own key. The engine could not tell a
complete list from one that hides the org's own voice, transcriber or
storage key.

### Current behavior (Verified)

- The state file records `name → uuid` for a credential and nothing else,
so the engine did not know which provider a saved key is for.
- `validate` only checked that each name on a list exists in the state file.
- Promotion dropped a `credentialIds` entry whose binding policy is `omit`
without saying so.
- Two credentials with no name and the same provider get the same slug on
pull, and the second silently replaces the first in the state file.

### Risk

An org that must run only on its own keys sends audio or prompts to Vapi's
provider accounts when one entry is missed in one squad file, with nothing
in the push output or the call to show it.

### Current mitigation

None before this change, other than reading every list by hand.

### Possible fix

Record each credential's provider on pull, report a list that leaves out a
key the org has saved for something the call uses, and let an org mark
itself customer-keys-only so the report is an error there.

### Status

**RESOLVED 2026-11-17.** Pull writes
`state/<org>/credential-providers.json`
(`resources/<org>/.vapi-credential-providers.json` while the org's state is
in the older layout) and warns when two
credentials collide on one name. `validate` has the
`credential-pin-hides-key` rule, an error in an org whose
`.vapi-policy.yml` sets `customerKeysOnly: true`. The promotion plan warns
when `omit` drops a `credentialIds` entry. See
[Credential pins](docs/guides/credential-pins.md).

Still open: the slug collision itself is only warned about, and the check
does not know which models or regions a key covers.

---

## Out of scope (intentionally not improvements)

- **State file is identity-only and not git-ignored.** It's intentionally
Expand Down
Loading
Loading