Skip to content

feat(validate): flag a credentialIds list that hides one of the org's keys - #96

Closed
dhruva-vapi wants to merge 9 commits into
feat/state-folderfrom
feat/credential-pin-check
Closed

dhruva-vapi wants to merge 9 commits into
feat/state-folderfrom
feat/credential-pin-check

Conversation

@dhruva-vapi

@dhruva-vapi dhruva-vapi commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor

Value

V.A.L.U.E. tier: small. One validate rule with the data it needs, a per-org switch, a promotion warning, tests and a guide.

  • Problem: a credentialIds list is the full set of keys a call may use, pooled across every squad member. A provider left off it doesn't fail. It runs on Vapi's own key, a tool request goes out without its key, and the recording skips the org's bucket. Nothing in the engine could tell a complete list from one that hides the org's own voice or transcriber key, and promotion dropped an omit entry from a list without saying so.
  • Who it affects: anyone who pins calls to specific keys, most of all an org that holds one model key per site and must run only on its own keys.

How it works

  1. Pull records what each key is for. The state file maps a credential's name to its UUID and nothing else, so the engine didn't know a key's provider. Pull now writes state/<org>/credential-providers.json (name → provider, plus whether a storage bucket is set). It holds no secret and is committed with the state file. While an org's state is in the older layout the file is resources/<org>/.vapi-credential-providers.json, which the commit step such a repository already has stages with the folder, and npm run migrate moves it with the rest. The state file format is unchanged.
  2. validate checks every list. The new credential-pin-hides-key finding fires when a list leaves out a key the org has saved for something the call uses: a model, voice or transcriber and their fallbacks, a key a tool or server names in credentialId, the storage bucket, and Langfuse. For a squad it looks at the call as a whole: every member's list pooled, and what each member uses after assistantOverrides and membersOverrides are applied.
  3. It leaves Vapi-default setups alone. A provider the org has no key of its own for is never reported. Another key for the same provider counts, so a squad pinned to one site's key isn't told about the other sites' keys. An OpenAI model is satisfied by an openai key or an azure-openai key.
  4. Warning by default, error by choice. customerKeysOnly: true in a new resources/<org>/.vapi-policy.yml makes the finding an error in that org, so validate fails, apply stops before it pulls or pushes, and the Validate resources check fails on the pull request. An unknown key in the policy file stops validate, so a typo can't switch the rule off.
  5. Promotion says what it drops. When the omit binding policy removes an entry from a credentialIds list, the plan names the entry and the file.
  6. Pull warns about colliding names. Two credentials with no name and the same provider get one slug, and the second replaced the first silently.

Guide: docs/guides/credential-pins.md.

Verification

Nothing here was run against a live org. Every behaviour is covered by tests against fixtures or a local stub API.

Rule Test
No list, no finding. A complete list passes tests/validate-credential-pins.test.ts
A hidden voice key is reported once for the squad, naming every member that uses it same
Error when the org is customer-keys-only, warning otherwise same
A provider the org has no key for is left alone same
Another key for the same provider counts. OpenAI model: openai or azure-openai same
membersOverrides replaces a member's own override list. The assistant's own list is kept and added to same
One member's list narrows the whole call same
Overrides that change a provider, fallback voices, tool and server credentialId, tools:append, storage, Langfuse, inline keys, UUID entries same
Missing or stale providers file asks for a pull instead of guessing same
Pull writes the providers file with no secret in it and leaves the state file as it was. validate then warns, fails once the policy is set, and passes once the key is listed. Run in both layouts, and the finding names the file the org has tests/credential-pins-cli.test.ts
Policy file parsing, and the providers file round trip. In the older layout the file sits under resources/<org>/ with nothing written under state/, and migrate moves it tests/credential-kinds.test.ts
Promotion warns for a dropped list entry, not for a single omitted credentialId, and not again once promoted tests/promotion.test.ts
In a fresh clone on the older layout, pull keeps a merged change while the providers record is still uncommitted: the record is the engine's file, not a person's edit tests/state-records-third-round.test.ts

Each rule was also checked by breaking it: six one-line mutations (no providers file written, lists replaced instead of joined, firing with no saved key, OpenAI limited to openai keys, override order swapped, dropped entries not collected) each fail the test that names the rule. Three more for where the file lives (always written under state/, left behind by migrate, the finding naming state/ whatever the layout) fail too.

npm run build passes and npm test passes 676/676 in CI (645 on the branch below), and npm run test:repository passes 9/9.

Things to know

  • An org that already uses credentialIds gets one new warning, credential-pin-unchecked, until it pulls once and commits the providers file. The upgrade notes say so.
  • A repository that stays on the older layout gets the file under resources/<org>/, so a commit step written before state/ existed commits it. Run end to end on such a repository with a recording stub of the API: the first CI run commits the file for every org, the requests are the same as main's, and validate in a clean clone afterwards reports no credential-pin-unchecked. An org with no lists sees nothing new except the new file after its next pull.
  • The check knows each key's provider, not which models or regions it covers. A list with an Azure OpenAI key that doesn't serve the assistant's model passes.
  • A squad member that isn't a file in the org's folder adds nothing to the check, and lists added at call time are outside the repo.
  • npm run push on its own keeps its existing behaviour: validation errors block it only with --strict.
  • The policy makes a hidden key an error. It doesn't require the org to have a key for every provider it uses.

Stack

Second of three. Sits on #97, which adds state/<org>/ and makes npm run migrate the only command that moves state there. The providers record follows whichever layout the org is in. #94 sits on top.

dhruva-vapi commented Nov 17, 2026 •

Copy link
Copy Markdown
Contributor Author

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

@dhruva-vapi
dhruva-vapi changed the base branch from main to graphite-base/96 November 17, 2026 21:30
@dhruva-vapi
dhruva-vapi force-pushed the feat/credential-pin-check branch from cf491b0 to bc80d57 Compare November 17, 2026 21:30
@dhruva-vapi
dhruva-vapi changed the base branch from graphite-base/96 to feat/state-folder November 17, 2026 21:30
@dhruva-vapi
dhruva-vapi force-pushed the feat/credential-pin-check branch 5 times, most recently from 0225d88 to 6c7b190 Compare November 19, 2026 13:34
@dhruva-vapi
dhruva-vapi marked this pull request as ready for review November 20, 2026 18:24
dhruva-vapi and others added 9 commits November 21, 2026 03:41
… keys

A credentialIds list is the full set of keys a call may use, pooled across
every squad member. A provider left off it runs on Vapi's own key with no
error. validate now reports a list that leaves out a key the org has saved
for something the call uses: a model, voice or transcriber (and fallbacks),
a key a tool or server names, the storage bucket and Langfuse. A provider
the org has no key of its own for is left alone.

The finding is a warning. It is an error in an org whose
resources/<org>/.vapi-policy.yml sets customerKeysOnly: true.

Pull now records each credential's provider in
resources/<org>/.vapi-credential-providers.json, since the state file only
maps names to UUIDs, and warns when two credentials collide on one name.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
An omitted credentialId is neutral. An entry omitted from a credentialIds
list narrows which keys calls in the target may use, so the provider it was
for falls back to Vapi's own key. The plan now names the entry and the file.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
The record is written by pull, so it belongs with the rest of the engine's
files. The hand-written .vapi-policy.yml stays in resources/<org>/.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…er an upgrade

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…ommit step stages it

An org whose state is still in the layout from before state/ gets its
record at resources/<org>/.vapi-credential-providers.json, beside the old
org pin. A commit step written for that layout stages resources/, so the
record is committed and validate stops reporting credential-pin-unchecked
on every run. An org whose state is in state/<org>/ keeps
state/<org>/credential-providers.json. npm run migrate moves the record
with the state file and the pin.

validateCredentialPins takes the file's name from its caller, so the
finding names the file the org actually has.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
In the older layout the record sits in resources/<org>/, and until it is
committed git shows it as untracked. Pull leaves the engine's own files out
of what it counts as a person's edits, and the record is on that list
because migrate moves it. This pins it with a pull in a fresh clone.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Every engine test file imports tests/engine-sandbox.ts first. These three
are the ones this branch adds.

Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
@devin-ai-integration

Copy link
Copy Markdown
Contributor

Replaced by #110 (same commits rebased onto main, opened as the Devin bot so it can be approved).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant