Repository navigation
feat(validate): flag a credentialIds list that hides one of the org's keys - #110
Open
devin-ai-integration[bot] wants to merge 9 commits into
Open
devin-ai-integration[bot] wants to merge 9 commits into
devin-ai-integration[bot] wants to merge 9 commits into
Conversation
… keys A credentialIds list is the full set of keys a call may use, pooled across every squad member. A provider left off it runs on Vapi's own key with no error. validate now reports a list that leaves out a key the org has saved for something the call uses: a model, voice or transcriber (and fallbacks), a key a tool or server names, the storage bucket and Langfuse. A provider the org has no key of its own for is left alone. The finding is a warning. It is an error in an org whose resources/<org>/.vapi-policy.yml sets customerKeysOnly: true. Pull now records each credential's provider in resources/<org>/.vapi-credential-providers.json, since the state file only maps names to UUIDs, and warns when two credentials collide on one name. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
An omitted credentialId is neutral. An entry omitted from a credentialIds list narrows which keys calls in the target may use, so the provider it was for falls back to Vapi's own key. The plan now names the entry and the file. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
The record is written by pull, so it belongs with the rest of the engine's files. The hand-written .vapi-policy.yml stays in resources/<org>/. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…er an upgrade Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
…ommit step stages it An org whose state is still in the layout from before state/ gets its record at resources/<org>/.vapi-credential-providers.json, beside the old org pin. A commit step written for that layout stages resources/, so the record is committed and validate stops reporting credential-pin-unchecked on every run. An org whose state is in state/<org>/ keeps state/<org>/credential-providers.json. npm run migrate moves the record with the state file and the pin. validateCredentialPins takes the file's name from its caller, so the finding names the file the org actually has. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
In the older layout the record sits in resources/<org>/, and until it is committed git shows it as untracked. Pull leaves the engine's own files out of what it counts as a person's edits, and the record is on that list because migrate moves it. This pins it with a pull in a fresh clone. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Every engine test file imports tests/engine-sandbox.ts first. These three are the ones this branch adds. Co-authored-by: capy-ai[bot] <230910855+capy-ai[bot]@users.noreply.github.com>
Co-Authored-By: Devin AI <158243242+devin-ai-integration[bot]@users.noreply.github.com>
Contributor
Author
|
I'll fix CI failures and address comments from users with write access that start with 'Devin'.
|
Contributor
Author
Devin Review🟢 Devin Review found nothing to fix · View findings |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Replaces #96 (same commits, rebased onto main), opened as the Devin bot so it can be approved.
Value
V.A.L.U.E. tier: small. One validate rule with the data it needs, a per-org switch, a promotion warning, tests and a guide.
credentialIdslist is the full set of keys a call may use, pooled across every squad member. A provider left off it doesn't fail. It runs on Vapi's own key, a tool request goes out without its key, and the recording skips the org's bucket. Nothing in the engine could tell a complete list from one that hides the org's own voice or transcriber key, and promotion dropped anomitentry from a list without saying so.How it works
state/<org>/credential-providers.json(name → provider, plus whether a storage bucket is set). It holds no secret and is committed with the state file. While an org's state is in the older layout the file isresources/<org>/.vapi-credential-providers.json, which the commit step such a repository already has stages with the folder, andnpm run migratemoves it with the rest. The state file format is unchanged.validatechecks every list. The newcredential-pin-hides-keyfinding fires when a list leaves out a key the org has saved for something the call uses: a model, voice or transcriber and their fallbacks, a key a tool or server names incredentialId, the storage bucket, and Langfuse. For a squad it looks at the call as a whole: every member's list pooled, and what each member uses afterassistantOverridesandmembersOverridesare applied.openaikey or anazure-openaikey.customerKeysOnly: truein a newresources/<org>/.vapi-policy.ymlmakes the finding an error in that org, sovalidatefails,applystops before it pulls or pushes, and the Validate resources check fails on the pull request. An unknown key in the policy file stopsvalidate, so a typo can't switch the rule off.omitbinding policy removes an entry from acredentialIdslist, the plan names the entry and the file.Guide:
docs/guides/credential-pins.md.Verification
Nothing here was run against a live org. Every behaviour is covered by tests against fixtures or a local stub API.
tests/validate-credential-pins.test.tsopenaiorazure-openaimembersOverridesreplaces a member's own override list. The assistant's own list is kept and added tocredentialId,tools:append, storage, Langfuse, inline keys, UUID entriesvalidatethen warns, fails once the policy is set, and passes once the key is listed. Run in both layouts, and the finding names the file the org hastests/credential-pins-cli.test.tsresources/<org>/with nothing written understate/, andmigratemoves ittests/credential-kinds.test.tscredentialId, and not again once promotedtests/promotion.test.tstests/state-records-third-round.test.tsEach rule was also checked by breaking it: six one-line mutations (no providers file written, lists replaced instead of joined, firing with no saved key, OpenAI limited to
openaikeys, override order swapped, dropped entries not collected) each fail the test that names the rule. Three more for where the file lives (always written understate/, left behind bymigrate, the finding namingstate/whatever the layout) fail too.npm run buildpasses andnpm testpasses 676/676 in CI (645 on the branch below), andnpm run test:repositorypasses 9/9.Things to know
credentialIdsgets one new warning,credential-pin-unchecked, until it pulls once and commits the providers file. The upgrade notes say so.resources/<org>/, so a commit step written beforestate/existed commits it. Run end to end on such a repository with a recording stub of the API: the first CI run commits the file for every org, the requests are the same as main's, andvalidatein a clean clone afterwards reports nocredential-pin-unchecked. An org with no lists sees nothing new except the new file after its next pull.npm run pushon its own keeps its existing behaviour: validation errors block it only with--strict.Stack
Second of three. Sits on #97, which adds
state/<org>/and makesnpm run migratethe only command that moves state there. The providers record follows whichever layout the org is in. #94 sits on top.