Skip to content

[AIR] CNXT-2686: warn when a pushed image sets WORKDIR - #7867

Open
vinchenzo-db wants to merge 5 commits into
mainfrom
vinchenzo-db/cnxt-2686
Open

vinchenzo-db wants to merge 5 commits into
mainfrom
vinchenzo-db/cnxt-2686

Conversation

@vinchenzo-db

@vinchenzo-db vinchenzo-db commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Changes

databricks air images push now reads Config.WorkingDir from the existing docker image inspect call (combined format string) and, when it is set to anything other than /, prints a warning that AI Runtime does not honor WORKDIR at runtime and that commands and scripts should use absolute paths. The push itself is unchanged: the warning is informational and never blocks.

Why

AI Runtime ignores the Docker WORKDIR directive, so an image whose entrypoint or scripts rely on relative paths works under docker run but fails on AIR with file-not-found errors. Until now the only mention of this was in the agent skill's docker-images notes, and neither air images push nor air run warned about it, so users only discovered the limitation from a failed run (Jira CNXT-2686, found in the CUJ PE64 runthrough).

Warning at push time is the earliest point where the CLI sees the image config, and it reuses the inspect call the platform check already makes, so there is no extra Docker invocation.

Tests

  • TestRunImagePushWarnsWhenImageSetsWorkingDir (table: unset, /, /app) checks the warning fires only for a real WORKDIR.
  • TestRunImagePushTagsAndPushesLocalImage asserts the combined inspect format and no warning for an image without WORKDIR.
  • Acceptance test acceptance/air/image-push gained a workdir fake-docker mode and a step showing the warning while the push proceeds; golden output regenerated.
  • go test ./cmd/air, go test ./acceptance -run TestAccept/air/image-push, task lint-q, and task check-changelog pass locally.

This pull request and its description were written by Isaac.

AI Runtime does not honor the Docker WORKDIR directive, so commands that
rely on relative paths fail at runtime with file-not-found errors, and
nothing warned at push time. `air images push` now inspects the source
image's Config.WorkingDir and prints a warning telling the user to use
absolute paths. The push itself is unchanged.

Co-authored-by: Isaac <no-reply@databricks.com>
@vinchenzo-db
vinchenzo-db requested review from a team as code owners November 21, 2026 04:11
Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions github-actions Bot added the AIR Databricks AI Runtime CLI label Nov 21, 2026
@vinchenzo-db

Copy link
Copy Markdown
Contributor Author

/review

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at 3031cd9. Each finding was verified against the code.

Not ready yet: 1 blocking issue(s), tagged [blocking].

This PR adds a WORKDIR warning to databricks air images push by inspecting the image config after the platform check. Two issues stand out: the "warning" can actually abort the push when the inspect subprocess fails, contradicting the stated non-blocking intent; and images that set WORKDIR / (effectively the default) trigger a misleading warning about relative paths.

Comment thread cmd/air/image.go Resolved
Comment thread cmd/air/image.go Resolved
An inspect failure now logs a warning instead of aborting, and WORKDIR /
(the Docker default) no longer triggers the warning.

Co-authored-by: Isaac <no-reply@databricks.com>
@vinchenzo-db

Copy link
Copy Markdown
Contributor Author

/review

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at 42e16c3. Each finding was verified against the code.

No blocking issues found. 1 non-blocking comment(s) below.

This PR adds an informational WORKDIR warning to databricks air images push by inspecting the image config after the platform check. The implementation is correct and well-tested, but the new check spawns a second docker image inspect subprocess on the same image that was just inspected, when a single call with a combined format string would suffice.

Comment thread cmd/air/image.go Resolved
Co-authored-by: Isaac <no-reply@databricks.com>
@vinchenzo-db

Copy link
Copy Markdown
Contributor Author

/review

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at d48fc7f. Each finding was verified against the code.

No blocking issues found.

This PR adds a WORKDIR warning to databricks air images push by extending the existing docker image inspect format string to also retrieve Config.WorkingDir, then printing an informational warning when a non-trivial WORKDIR is set. The change is small, well-scoped, and covered by new and updated unit tests. No correctness issues survive verification.

Co-authored-by: Isaac <no-reply@databricks.com>
@vinchenzo-db

Copy link
Copy Markdown
Contributor Author

/review

@github-actions

Copy link
Copy Markdown
Contributor

Commit: 0f620d56e | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0
test branch main (a4388d7) latest (v1.27.0)
TestAccept/air/image-push ✅ ❌ ❌
TestAccept/air/image-push: branch ✅ | main (a4388d7) ❌ | latest (v1.27.0) ❌

main (a4388d7):

acceptance_test.go:1402: Diff:
        --- air/image-push/output.txt
        +++ /tmp/TestAcceptairimage-push2388300117/002/output.txt
        @@ -8,252 +8,6 @@
         Updated Docker config: [TEST_TMP_DIR]/home/docker/config.json
         Installed Docker credential helper: [TEST_TMP_DIR]/home/bin/docker-credential-databricks
         Using local source image nvidia/cuda:13.4.1
        -Tagging image as [WORKSPACE_ID].container.us-west-2.localhost/my-catalog.my-schema.cuda:13.4.1-cudnn-devel-ubuntu26.04
        -Pushing image to [WORKSPACE_ID].container.us-west-2.localhost/my-catalog.my-schema.cuda:13.4.1-cudnn-devel-ubuntu26.04
        -Docker push progress
        -Pushed image.
        -Now you can set `environment.unity_catalog_image = my-catalog.my-schema.cuda:13.4.1-cudnn-devel-ubuntu26.04` in your YAML config and create a workload with `databricks air run -f config.yaml`.
        +Error: source image nvidia/cuda:13.4.1 has platform "linux/amd64|"; AIR requires linux/amd64: use --pull to fetch a compatible image, or rebuild the source for linux/amd64
         
        -=== inferred region and workspace ID requests
        -
        ->>> print_requests.py --get //api
        -{
        -  "method": "GET",
        -  "path": "/api/2.0/preview/scim/v2/Me",
        -  "q": {
        -    "excludedAttributes": "entitlements"
        -  }
        -}
        -{
        -  "method": "GET",
        -  "path": "/api/2.1/unity-catalog/metastore_summary"
        -}
        -
        -=== docker commands
        -image ls --quiet nvidia/cuda:13.4.1
        -image inspect --format {{.Os}}/{{.Architecture}}|{{.Config.WorkingDir}} nvidia/cuda:13.4.1
        -tag nvidia/cuda:13.4.1 [WORKSPACE_ID].container.us-west-2.localhost/my-catalog.my-schema.cuda:13.4.1-cudnn-devel-ubuntu26.04
        -push [WORKSPACE_ID].container.us-west-2.localhost/my-catalog.my-schema.cuda:13.4.1-cudnn-devel-ubuntu26.04
        -
        -=== missing image is pulled for AIR; destination defaults to source
        -
        ->>> databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training
        -Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
        -Pushing example/image:v1 as main.training.image:v1 using profile docker-test
        -Configured Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost
        -Updated Docker config: [TEST_TMP_DIR]/home/docker/config.json
        -Installed Docker credential helper: [TEST_TMP_DIR]/home/bin/docker-credential-databricks
        -Pulling source image example/image:v1 for linux/amd64
        -Docker pull progress
        -Tagging image as [WORKSPACE_ID].container.us-west-2.localhost/main.training.image:v1
        -Pushing image to [WORKSPACE_ID].container.us-west-2.localhost/main.training.image:v1
        -Docker push progress
        -Pushed image.
        -Now you can set `environment.unity_catalog_image = main.training.image:v1` in your YAML config and create a workload with `databricks air run -f config.yaml`.
        -
        -=== forced pull and JSON output
        -
        ->>> databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training --pull -o json
        -Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
        -Pushing example/image:v1 as main.training.image:v1 using profile docker-test
        -Configured Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost
        -Updated Docker config: [TEST_TMP_DIR]/home/docker/config.json
        -Installed Docker credential helper: [TEST_TMP_DIR]/home/bin/docker-credential-databricks
        -Pulling source image example/image:v1 for linux/amd64
        -Docker pull progress
        -Tagging image as [WORKSPACE_ID].container.us-west-2.localhost/main.training.image:v1
        -Pushing image to [WORKSPACE_ID].container.us-west-2.localhost/main.training.image:v1
        -Docker push progress
        -Pushed image.
        -
        ->>> jq -e {v, data} [TEST_TMP_DIR]/home/result.json
        -{
        -  "v": 1,
        -  "data": {
        -    "unity_catalog_image": "main.training.image:v1",
        -    "registry_image": "[WORKSPACE_ID].container.us-west-2.localhost/main.training.image:v1"
        -  }
        -}
        -
        -=== image with WORKDIR warns but is pushed
        -
        ->>> databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training
        -Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
        -Pushing example/image:v1 as main.training.image:v1 using profile docker-test
        -Configured Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost
        -Updated Docker config: [TEST_TMP_DIR]/home/docker/config.json
        -Installed Docker credential helper: [TEST_TMP_DIR]/home/bin/docker-credential-databricks
        -Using local source image example/image:v1
        -Warning: source image example/image:v1 sets WORKDIR /app, which AI Runtime does not honor at runtime: use absolute paths in your command and scripts.
        -Tagging image as [WORKSPACE_ID].container.us-west-2.localhost/main.training.image:v1
        -Pushing image to [WORKSPACE_ID].container.us-west-2.localhost/main.training.image:v1
        -Docker push progress
        -Pushed image.
        -Now you can set `environment.unity_catalog_image = main.training.image:v1` in your YAML config and create a workload with `databricks air run -f config.yaml`.
        -
        -=== daemon failure must not pull
        -
        ->>> musterr databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training
        -Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
        -Pushing example/image:v1 as main.training.image:v1 using profile docker-test
        -Configured Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost
        -Updated Docker config: [TEST_TMP_DIR]/home/docker/config.json
        -Installed Docker credential helper: [TEST_TMP_DIR]/home/bin/docker-credential-databricks
        -Cannot connect to the Docker daemon
        -Error: check local source image example/image:v1 (ensure Docker is running and accessible): exit status 1
        -
        -=== inspect failure must not pull
        -
        ->>> musterr databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training
        -Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
        -Pushing example/image:v1 as main.training.image:v1 using profile docker-test
        -Configured Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost
        -Updated Docker config: [TEST_TMP_DIR]/home/docker/config.json
        -Installed Docker credential helper: [TEST_TMP_DIR]/home/bin/docker-credential-databricks
        -Using local source image example/image:v1
        -Docker inspect failed
        -Error: inspect source image example/image:v1: exit status 1
        -
        -=== incompatible local image must not be pushed
        -
        ->>> musterr databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training
        -Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
        -Pushing example/image:v1 as main.training.image:v1 using profile docker-test
        -Configured Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost
        -Updated Docker config: [TEST_TMP_DIR]/home/docker/config.json
        -Installed Docker credential helper: [TEST_TMP_DIR]/home/bin/docker-credential-databricks
        -Using local source image example/image:v1
        -Error: source image example/image:v1 has platform "linux/arm64"; AIR requires linux/amd64: use --pull to fetch a compatible image, or rebuild the source for linux/amd64
        -
        -=== incompatible pulled image must not be pushed
        -
        ->>> musterr databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training --pull
        -Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
        -Pushing example/image:v1 as main.training.image:v1 using profile docker-test
        -Configured Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost
        -Updated Docker config: [TEST_TMP_DIR]/home/docker/config.json
        -Installed Docker credential helper: [TEST_TMP_DIR]/home/bin/docker-credential-databricks
        -Pulling source image example/image:v1 for linux/amd64
        -Docker pull progress
        -Error: source image example/image:v1 has platform "linux/arm64"; AIR requires linux/amd64: use --pull to fetch a compatible image, or rebuild the source for linux/amd64
        -
        -=== pull failure must not be pushed
        -
        ->>> musterr databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training --pull
        -Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
        -Pushing example/image:v1 as main.training.image:v1 using profile docker-test
        -Configured Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost
        -Updated Docker config: [TEST_TMP_DIR]/home/docker/config.json
        -Installed Docker credential helper: [TEST_TMP_DIR]/home/bin/docker-credential-databricks
        -Pulling source image example/image:v1 for linux/amd64
        -Docker pull failed
        -Error: pull source image example/image:v1: exit status 1
        -
        -=== invalid destination and removed flags
        -
        ->>> musterr databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training --artifact invalid.name
        -Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
        -Error: invalid artifact name "invalid.name": use lowercase letters and digits separated by one or two underscores or one or more hyphens
        -
        ->>> musterr databricks air images push -p docker-test -s example/image:v1 --catalog main --schema team_-training
        -Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
        -Error: invalid schema "team_-training": use lowercase letters and digits separated by one or two underscores or one or more hyphens
        -
        ->>> musterr databricks air images push --region us-west-2
        -Error: unknown flag: --region
        -
        -Usage:
        -  databricks air images push [flags]
        -
        -Flags:
        -      --artifact string     Destination ARTIFACT[:TAG]; defaults to the source image name and tag (latest if untagged); maximum 255 characters for catalog.schema.artifact and 128 for the tag
        -      --catalog string      Destination Unity Catalog catalog; use lowercase letters and digits separated by one or two underscores or one or more hyphens
        -  -h, --help[=true|false]   help for push
        -      --pull[=true|false]   Pull the source image even when it is already available locally
        -      --schema string       Destination Unity Catalog schema; use lowercase letters and digits separated by one or two underscores or one or more hyphens
        -  -s, --source string       Source container image to push (NAME[:TAG] or NAME@DIGEST)
        -
        -Global Flags:
        -      --debug[=true|false]   enable debug logging
        -  -o, --output type          output type: text or json (default text)
        -  -p, --profile string       ~/.databrickscfg profile
        -  -t, --target string        bundle target to use (if applicable)
        -
        -
        ->>> musterr databricks air images push --image artifact
        -Error: unknown flag: --image
        -
        -Usage:
        -  databricks air images push [flags]
        -
        -Flags:
        -      --artifact string     Destination ARTIFACT[:TAG]; defaults to the source image name and tag (latest if untagged); maximum 255 characters for catalog.schema.artifact and 128 for the tag
        -      --catalog string      Destination Unity Catalog catalog; use lowercase letters and digits separated by one or two underscores or one or more hyphens
        -  -h, --help[=true|false]   help for push
        -      --pull[=true|false]   Pull the source image even when it is already available locally
        -      --schema string       Destination Unity Catalog schema; use lowercase letters and digits separated by one or two underscores or one or more hyphens
        -  -s, --source string       Source container image to push (NAME[:TAG] or NAME@DIGEST)
        -
        -Global Flags:
        -      --debug[=true|false]   enable debug logging
        -  -o, --output type          output type: text or json (default text)
        -  -p, --profile string       ~/.databrickscfg profile
        -  -t, --target string        bundle target to use (if applicable)
        -
... (60 more lines truncated, raise DATABRICKS_CLI_TEST_MAX_LOG to see more)

latest (v1.27.0): same output as main (a4388d7)

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at 0f620d5. Each finding was verified against the code.

No blocking issues found.

This PR adds an informational warning to databricks air images push when the source image sets a non-default WORKDIR, since AI Runtime ignores it at runtime. The implementation correctly extends the existing docker image inspect format string with a pipe-delimited field and parses it with strings.Cut on the first |. The warning condition (workingDir != "" && workingDir != "/") is sound, the comment on line 209 explains the rationale, and the unit and acceptance tests cover the key cases (unset, /, /app). No correctness issues found.

@eng-dev-ecosystem-bot

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 0f620d5

Run: 44867201147

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 20 4:33
💚​ aws-windows-latest-4core-16gb 1 274 18 3:57
💚​ azure linux-2core-8gb 1 271 20 5:01
💚​ azure-windows-latest-4core-16gb 1 273 18 4:22
💚​ gcp linux-2core-8gb 1 272 20 5:44
💚​ gcp-windows-latest-4core-16gb 1 274 18 3:13
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 3 slowest tests (at least 2 minutes):
duration env testname
4:20 azure-windows-latest-4core-16gb TestAccept
3:55 aws-windows-latest-4core-16gb TestAccept
3:12 gcp-windows-latest-4core-16gb TestAccept

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIR Databricks AI Runtime CLI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants