Skip to content

[AIR] CNXT-2683: Emit air run config-validation errors as JSON with -o json - #7864

Open
vinchenzo-db wants to merge 4 commits into
mainfrom
vinchenzo-db/cnxt-2683
Open

vinchenzo-db wants to merge 4 commits into
mainfrom
vinchenzo-db/cnxt-2683

Conversation

@vinchenzo-db

@vinchenzo-db vinchenzo-db commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Changes

databricks air run -o json now reports config-validation failures through the same JSON error envelope the other air commands use ({"v": 1, "ts": ..., "error": {"code": "INVALID_ARGS", "kind": "PERMANENT", "message": ..., "retryable": false}}) instead of plain text. This covers every validation step before a run is submitted: the local schema validation of the YAML config and --override values (for example max_retries must be >= 0, got -1), code_source snapshot resolution, the idempotency-token length check, and the backend field errors returned by --dry-run. A --dry-run that cannot build an authenticated workspace client is reported as UNAUTHENTICATED through the existing authError helper, like the other air commands.

Why

Scripts and coding agents that pass -o json expect every outcome on stdout to be a JSON envelope. Config-validation errors were the one air run failure still printed as Error: ... text, which broke those consumers (Jira CNXT-2683, found during the CUJ PE64 runthrough). The fix reuses the existing renderError helper, so the envelope shape is unchanged; text mode is unaffected.

The submitWorkload error on the non-dry-run path is left as is: it mixes artifact-upload and API failures with preflight validation, and none of the existing envelope codes describes all of them. Classifying those errors is a separate change.

Tests

  • Added -o json cases to the acceptance/air/run ("invalid config is rejected (json)", "include_paths entry that selects no files is rejected (json)", "dry-run without credentials reports an authentication error (json)") and acceptance/air/run-backend-error ("backend field errors fail dry-run (json)") acceptance tests; regenerated their outputs with -update.
  • go test ./acceptance -run TestAccept/air/ passes. task lint-q and task ws are clean.

This pull request and its description were written by Isaac.

vinchenzo-db and others added 2 commits November 21, 2026 03:37
`databricks air run -o json` printed config-validation failures (local
schema errors such as "max_retries must be >= 0" and backend field errors
from --dry-run) as plain text, so scripts parsing the JSON envelope broke.
Route both through the existing error envelope (INVALID_ARGS / PERMANENT),
matching the other air commands.

Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions github-actions Bot added the AIR Databricks AI Runtime CLI label Nov 21, 2026
@vinchenzo-db
vinchenzo-db marked this pull request as ready for review November 21, 2026 03:38
@vinchenzo-db
vinchenzo-db requested review from a team as code owners November 21, 2026 03:38
@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 4d9b33d5d | Workflow run

Regression test report

Selected 2 changed tests (limit=20, 0 not selected)

2 tests: regression=2 unreleased=0 coverage=0 failing=0
test branch main (1f1ab66) latest (v1.27.0)
TestAccept/air/run ✅ ❌ ❌
TestAccept/air/run-backend-error ✅ ❌ ❌
TestAccept/air/run: branch ✅ | main (1f1ab66) ❌ | latest (v1.27.0) ❌

main (1f1ab66):

acceptance_test.go:1402: Diff:
        --- air/run/output.txt
        +++ /tmp/TestAcceptairrun1543020101/002/output.txt
        @@ -66,17 +66,11 @@
         === include_paths entry that selects no files is rejected (json)
         
         >>> [CLI] air run -f include-unmatched.yaml --dry-run -o json
        -{
        -  "v": 1,
        -  "ts": "[TIMESTAMP]",
        -  "error": {
        -    "code": "INVALID_ARGS",
        -    "kind": "PERMANENT",
        -    "message": "code_source include_paths select no files under [TEST_TMP_DIR]: data. Check that these paths exist under root_path and hold files git does not ignore (`git check-ignore -v \u003cfile\u003e` shows the rule for a file); empty directories, nested git repositories and submodules add no files",
        -    "retryable": false
        -  }
        -}
        +Error: code_source include_paths select no files under [TEST_TMP_DIR]: data. Check that these paths exist under root_path and hold files git does not ignore (`git check-ignore -v <file>` shows the rule for a file); empty directories, nested git repositories and submodules add no files
         
        +Tip: use the --debug flag to see more details and a trace of this error:
        +  databricks --debug air run …
        +
         === include_paths pathspec that selects files outside root_path is rejected
         
         >>> [CLI] air run -f include-outside-root.yaml --dry-run
        @@ -137,31 +130,19 @@
         === invalid config is rejected (json)
         
         >>> [CLI] air run -f invalid.yaml --dry-run -o json
        -{
        -  "v": 1,
        -  "ts": "[TIMESTAMP]",
        -  "error": {
        -    "code": "INVALID_ARGS",
        -    "kind": "PERMANENT",
        -    "message": "invalid experiment_name \"bad.name\": only alphanumeric characters, hyphens (-), and underscores (_) are allowed",
        -    "retryable": false
        -  }
        -}
        +Error: invalid experiment_name "bad.name": only alphanumeric characters, hyphens (-), and underscores (_) are allowed
         
        +Tip: use the --debug flag to see more details and a trace of this error:
        +  databricks --debug air run …
        +
         === dry-run without credentials reports an authentication error (json)
         
         >>> env -u DATABRICKS_HOST -u DATABRICKS_TOKEN [CLI] air run -f valid.yaml --dry-run -o json
        -{
        -  "v": 1,
        -  "ts": "[TIMESTAMP]",
        -  "error": {
        -    "code": "UNAUTHENTICATED",
        -    "kind": "PERMANENT",
        -    "message": "no default profile is set: pass --profile (-p) or configure a default profile in your .databrickscfg",
        -    "retryable": false
        -  }
        -}
        +Error: default auth: cannot configure default credentials, please check https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication to configure credentials for your preferred authentication method. Config: databricks_cli_path=[CLI]. Env: DATABRICKS_CLI_PATH
         
        +Tip: use the --debug flag to see more details and a trace of this error:
        +  databricks --debug air run …
        +
         === unknown field suggests the near match
         
         >>> [CLI] air run -f unknown-field.yaml --dry-run
        
FAIL

latest (v1.27.0):

acceptance_test.go:1402: Diff:
        --- air/run/output.txt
        +++ /tmp/TestAcceptairrun3557449897/002/output.txt
        @@ -66,17 +66,11 @@
         === include_paths entry that selects no files is rejected (json)
         
         >>> [CLI] air run -f include-unmatched.yaml --dry-run -o json
        -{
        -  "v": 1,
        -  "ts": "[TIMESTAMP]",
        -  "error": {
        -    "code": "INVALID_ARGS",
        -    "kind": "PERMANENT",
        -    "message": "code_source include_paths select no files under [TEST_TMP_DIR]: data. Check that these paths exist under root_path and hold files git does not ignore (`git check-ignore -v \u003cfile\u003e` shows the rule for a file); empty directories, nested git repositories and submodules add no files",
        -    "retryable": false
        -  }
        -}
        +Error: code_source include_paths select no files under [TEST_TMP_DIR]: data. Check that these paths exist under root_path and hold files git does not ignore (`git check-ignore -v <file>` shows the rule for a file); empty directories, nested git repositories and submodules add no files
         
        +Tip: use the --debug flag to see more details and a trace of this error:
        +  databricks --debug air run …
        +
         === include_paths pathspec that selects files outside root_path is rejected
         
         >>> [CLI] air run -f include-outside-root.yaml --dry-run
        @@ -137,31 +130,19 @@
         === invalid config is rejected (json)
         
         >>> [CLI] air run -f invalid.yaml --dry-run -o json
        -{
        -  "v": 1,
        -  "ts": "[TIMESTAMP]",
        -  "error": {
        -    "code": "INVALID_ARGS",
        -    "kind": "PERMANENT",
        -    "message": "invalid experiment_name \"bad.name\": only alphanumeric characters, hyphens (-), and underscores (_) are allowed",
        -    "retryable": false
        -  }
        -}
        +Error: invalid experiment_name "bad.name": only alphanumeric characters, hyphens (-), and underscores (_) are allowed
         
        +Tip: use the --debug flag to see more details and a trace of this error:
        +  databricks --debug air run …
        +
         === dry-run without credentials reports an authentication error (json)
         
         >>> env -u DATABRICKS_HOST -u DATABRICKS_TOKEN [CLI] air run -f valid.yaml --dry-run -o json
        -{
        -  "v": 1,
        -  "ts": "[TIMESTAMP]",
        -  "error": {
        -    "code": "UNAUTHENTICATED",
        -    "kind": "PERMANENT",
        -    "message": "no default profile is set: pass --profile (-p) or configure a default profile in your .databrickscfg",
        -    "retryable": false
        -  }
        -}
        +Error: default auth: cannot configure default credentials, please check https://docs.databricks.com/en/dev-tools/auth.html#databricks-client-unified-authentication to configure credentials for your preferred authentication method. Config: databricks_cli_path=[CLI]. Env: DATABRICKS_CLI_PATH
         
        +Tip: use the --debug flag to see more details and a trace of this error:
        +  databricks --debug air run …
        +
         === unknown field suggests the near match
         
         >>> [CLI] air run -f unknown-field.yaml --dry-run
        @@ -189,23 +165,8 @@
         === removed pool field is rejected
         
         >>> [CLI] air run -f removed-pool.yaml --dry-run
        -Error: invalid config removed-pool.yaml: line 6: unknown field "compute.pool_id"; did you mean "accelerator_pool_id"? Run "databricks air run -h config.compute" for the list of fields
        +Dry run: configuration for "smoke-test" is valid; not submitting.
         
        -Tip: use the --debug flag to see more details and a trace of this error:
        -  databricks --debug air run …
        +Unexpected success
         
        -=== unsupported priority is rejected
        -
        ->>> [CLI] air run -f invalid-priority.yaml --dry-run
        -Error: compute.priority_class: invalid priority_class "HIGH": valid values are: NORMAL
        -
        -Tip: use the --debug flag to see more details and a trace of this error:
        -  databricks --debug air run …
        -
        -=== priority without a pool is rejected
        -
        ->>> [CLI] air run -f priority-without-pool.yaml --dry-run
        -Error: compute.priority_class requires compute.accelerator_pool_id — priority applies only within a GPU Pool
        -
        -Tip: use the --debug flag to see more details and a trace of this error:
        -  databricks --debug air run …
        +Exit code: 1
        
FAIL
TestAccept/air/run-backend-error: branch ✅ | main (1f1ab66) ❌ | latest (v1.27.0) ❌

main (1f1ab66):

acceptance_test.go:1402: 
        	Error Trace:	/home/runner/work/cli/cli/libs/testdiff/testdiff.go:22
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1402
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:1143
        	            				/home/runner/work/cli/cli/acceptance/acceptance_test.go:686
        	Error:      	Not equal: 
        	            	expected: "\n=== backend field errors fail dry-run\n\n>>> [CLI] air run -f valid.yaml --dry-run\nError: config validation failed:\n  deployments[0].compute.accelerator_count: accelerator_count must be at most 1024, but was 1032.\n  env_variables[1INVALID]: env_variables[1INVALID] contains characters that are not allowed.\n\nTip: use the --debug flag to see more details and a trace of this error:\n  databricks --debug air run …\n\n=== backend field errors fail dry-run (json)\n\n>>> [CLI] air run -f valid.yaml --dry-run -o json\n{\n  \"v\": 1,\n  \"ts\": \"[TIMESTAMP]\",\n  \"error\": {\n    \"code\": \"INVALID_ARGS\",\n    \"kind\": \"PERMANENT\",\n    \"message\": \"config validation failed:\\n  deployments[0].compute.accelerator_count: accelerator_count must be at most 1024, but was 1032.\\n  env_variables[1INVALID]: env_variables[1INVALID] contains characters that are not allowed.\",\n    \"retryable\": false\n  }\n}\n"
        	            	actual  : "\n=== backend field errors fail dry-run\n\n>>> [CLI] air run -f valid.yaml --dry-run\nError: config validation failed:\n  deployments[0].compute.accelerator_count: accelerator_count must be at most 1024, but was 1032.\n  env_variables[1INVALID]: env_variables[1INVALID] contains characters that are not allowed.\n\nTip: use the --debug flag to see more details and a trace of this error:\n  databricks --debug air run …\n\n=== backend field errors fail dry-run (json)\n\n>>> [CLI] air run -f valid.yaml --dry-run -o json\nError: config validation failed:\n  deployments[0].compute.accelerator_count: accelerator_count must be at most 1024, but was 1032.\n  env_variables[1INVALID]: env_variables[1INVALID] contains characters that are not allowed.\n\nTip: use the --debug flag to see more details and a trace of this error:\n  databricks --debug air run …\n"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -14,12 +14,8 @@
        	            	 >>> [CLI] air run -f valid.yaml --dry-run -o json
        	            	-{
        	            	-  "v": 1,
        	            	-  "ts": "[TIMESTAMP]",
        	            	-  "error": {
        	            	-    "code": "INVALID_ARGS",
        	            	-    "kind": "PERMANENT",
        	            	-    "message": "config validation failed:\n  deployments[0].compute.accelerator_count: accelerator_count must be at most 1024, but was 1032.\n  env_variables[1INVALID]: env_variables[1INVALID] contains characters that are not allowed.",
        	            	-    "retryable": false
        	            	-  }
        	            	-}
        	            	+Error: config validation failed:
        	            	+  deployments[0].compute.accelerator_count: accelerator_count must be at most 1024, but was 1032.
        	            	+  env_variables[1INVALID]: env_variables[1INVALID] contains characters that are not allowed.
        	            	 
        	            	+Tip: use the --debug flag to see more details and a trace of this error:
        	            	+  databricks --debug air run …
        	            	+
        	Test:       	TestAccept/air/run-backend-error
        	Messages:   	air/run-backend-error/output.txt vs /tmp/TestAcceptairrun-backend-error3767610864/002/output.txt

latest (v1.27.0): same output as main (1f1ab66)

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at bed61e9. Each finding was verified against the code.

Not ready yet: 1 blocking issue(s), tagged [blocking]; part of the automated review did not complete.

This PR wraps two air run error paths (loadRunConfigWithOverrides and backend-validation errors) with renderError so they emit JSON envelopes under -o json. The fix is correct for those two sites, but the same RunE function has additional bare return err paths—inside the dryRun block it already modifies—that still emit plain text in JSON mode. The new acceptance test also uses errcode where the testing rules require musterr.

Comments outside the changed lines

  • [blocking] cmd/air/run.go:178: countSnapshotFiles (line 178) and validateIdempotencyToken (line 185) are inside the same dryRun block where lines 165 and 190 now call renderError, but these two still do a bare return err. With --dry-run -o json and a config that has code_source.snapshot pointing at a missing directory, or an invalid idempotency token, the CLI prints plain Error: … instead of a JSON envelope—the exact class of bug this PR fixes elsewhere. submitWorkload at line 221 has the same gap on the non-dry-run path. Wrap all three in renderError (it already no-ops in text mode), or add a single fallback wrapper at the RunE boundary so future error paths are covered automatically.

Comment thread acceptance/air/run/script Resolved
…ON test

Review round 1: code_source snapshot resolution and the idempotency-token
check in the --dry-run block also returned plain text under -o json. Route
them through renderError too, and assert the expected failure with musterr.

Co-authored-by: Isaac <no-reply@databricks.com>
@vinchenzo-db

Copy link
Copy Markdown
Contributor Author

/review

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at 71adc65. Each finding was verified against the code.

Not ready yet: 1 blocking issue(s), tagged [blocking].

This PR makes air run -o json emit config-validation failures as structured JSON error envelopes instead of plain text, matching the other air commands. The change wraps four error-return sites in renderError calls. One of these sites can receive authentication errors that get mislabeled; otherwise the approach is consistent with the existing air get pattern.

Comment thread cmd/air/run.go Resolved
Review round 2: a failed workspace-client setup during --dry-run was
labeled INVALID_ARGS. Mark it as an authentication error and route it
through authError, like the other air commands.

Co-authored-by: Isaac <no-reply@databricks.com>
@vinchenzo-db

Copy link
Copy Markdown
Contributor Author

/review

@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 4d9b33d

Run: 44864660603

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 21 4:11
💚​ aws-windows-latest-4core-16gb 1 274 19 3:40
💚​ azure linux-2core-8gb 1 271 21 4:26
💚​ azure-windows-latest-4core-16gb 1 273 19 3:11
💚​ gcp linux-2core-8gb 1 272 21 3:49
💚​ gcp-windows-latest-4core-16gb 1 274 19 3:22
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
3:39 aws-windows-latest-4core-16gb TestAccept
3:15 gcp-windows-latest-4core-16gb TestAccept
3:10 azure-windows-latest-4core-16gb TestAccept

@vinchenzo-db

Copy link
Copy Markdown
Contributor Author

/review

1 similar comment
@vinchenzo-db

Copy link
Copy Markdown
Contributor Author

/review

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIR Databricks AI Runtime CLI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants