Skip to content

[AIR] CNXT-2687: print the source image size and warn about the 30 GB registry limit before pushing - #7868

Open
vinchenzo-db wants to merge 6 commits into
mainfrom
vinchenzo-db/cnxt-2687
Open

vinchenzo-db wants to merge 6 commits into
mainfrom
vinchenzo-db/cnxt-2687

Conversation

@vinchenzo-db

@vinchenzo-db vinchenzo-db commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Changes

databricks air images push now reads {{.Size}} from the existing docker image inspect call (combined format string, no extra Docker process) and prints the size (Source image X is 7.1 GB uncompressed; Databricks Artifact Registry caps images at 30 GB compressed.); if the size cannot be read, it warns with the raw inspect value instead. When the local image exceeds 30 GB it also prints a warning that Databricks Artifact Registry may reject the push, with one line of advice: start from a Databricks base image (databricksruntime/air) and trim layers such as model weights and build caches. The size is advisory: an oversized or unparsable size only warns, and the push proceeds. The upload itself is unchanged.

Why

Databricks Artifact Registry rejects images over 30 GB, but the CLI gave no size information before the upload, so a multi-gigabyte push from a laptop could run for a long time before failing (Jira CNXT-2687, found in the CUJ PE64 runthrough). Docker already knows the local image size, so printing it before the push costs nothing extra and lets the user stop early. AI Runtime itself no longer limits image size, so the PR only refers to the registry's limit.

Supervisor's call: warn, not fail, because the local size is uncompressed while the registry measures compressed layers, so a hard local failure would block images the registry would accept. The size uses decimal gigabytes, matching Docker's docker images output.

Tests

  • TestRunImagePushWarnsAboutOversizedImage (table: at limit, over limit, unreadable size) checks that an image at exactly 30 GB prints no warning, an oversized one prints the warning and is still pushed, and an unparsable size prints a warning and is still pushed.
  • TestRunImagePushTagsAndPushesLocalImage asserts the combined inspect format and the printed size line.
  • Acceptance test acceptance/air/image-push gained an oversized fake-docker mode and an "oversized image warns but is pushed" step; golden output regenerated.
  • go test ./cmd/air -run 'TestRunImagePush|TestImageExistsLocally', go test ./acceptance -run TestAccept/air/image-push, task lint-q, task ws and task check-changelog pass locally. Two unrelated cmd/air tests (TestGetRunMLflowRetryJSON, TestLogsCommandRepeatsRunLinkAfterStreaming) fail the same way on clean main in this sandbox.

This pull request and its description were written by Isaac.

`databricks air images push` now inspects the local image size after the
platform check, prints it, and fails before tagging or uploading when it
exceeds AI Runtime's 20 GB maximum, with the server's message plus advice
to start from a Databricks base image and trim layers (CNXT-2687).

Co-authored-by: Isaac <no-reply@databricks.com>
@vinchenzo-db
vinchenzo-db requested review from a team as code owners November 21, 2026 05:21
Co-authored-by: Isaac <no-reply@databricks.com>
@vinchenzo-db

Copy link
Copy Markdown
Contributor Author

/review

@github-actions github-actions Bot added the AIR Databricks AI Runtime CLI label Nov 21, 2026
Docker reports the uncompressed size while the server measures compressed
layers, so an oversized local image may still be accepted: print the size,
warn with the limit and advice, and let the push proceed.

Co-authored-by: Isaac <no-reply@databricks.com>
@vinchenzo-db

Copy link
Copy Markdown
Contributor Author

/review

@vinchenzo-db vinchenzo-db changed the title [AIR] CNXT-2687: check the source image size against the 20 GB limit before pushing [AIR] CNXT-2687: print the source image size and warn about the 20 GB limit before pushing Nov 21, 2026
@github-actions

github-actions Bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Contributor

Commit: 47a3dedc8 | Workflow run

Regression test report

Selected 1 changed tests (limit=20, 0 not selected)

1 tests: regression=1 unreleased=0 coverage=0 failing=0
test branch main (a4388d7) latest (v1.27.0)
TestAccept/air/image-push ✅ ❌ ❌
TestAccept/air/image-push: branch ✅ | main (a4388d7) ❌ | latest (v1.27.0) ❌

main (a4388d7):

acceptance_test.go:1402: Diff:
        --- air/image-push/output.txt
        +++ /tmp/TestAcceptairimage-push4242318683/002/output.txt
        @@ -8,257 +8,6 @@
         Updated Docker config: [TEST_TMP_DIR]/home/docker/config.json
         Installed Docker credential helper: [TEST_TMP_DIR]/home/bin/docker-credential-databricks
         Using local source image nvidia/cuda:13.4.1
        -Source image nvidia/cuda:13.4.1 is 7.1 GB uncompressed; Databricks Artifact Registry caps images at 30 GB compressed.
        -Tagging image as [WORKSPACE_ID].container.us-west-2.localhost/my-catalog.my-schema.cuda:13.4.1-cudnn-devel-ubuntu26.04
        -Pushing image to [WORKSPACE_ID].container.us-west-2.localhost/my-catalog.my-schema.cuda:13.4.1-cudnn-devel-ubuntu26.04
        -Docker push progress
        -Pushed image.
        -Now you can set `environment.unity_catalog_image = my-catalog.my-schema.cuda:13.4.1-cudnn-devel-ubuntu26.04` in your YAML config and create a workload with `databricks air run -f config.yaml`.
        +Error: source image nvidia/cuda:13.4.1 has platform "linux/amd64|7140000000"; AIR requires linux/amd64: use --pull to fetch a compatible image, or rebuild the source for linux/amd64
         
        -=== inferred region and workspace ID requests
        -
        ->>> print_requests.py --get //api
        -{
        -  "method": "GET",
        -  "path": "/api/2.0/preview/scim/v2/Me",
        -  "q": {
        -    "excludedAttributes": "entitlements"
        -  }
        -}
        -{
        -  "method": "GET",
        -  "path": "/api/2.1/unity-catalog/metastore_summary"
        -}
        -
        -=== docker commands
        -image ls --quiet nvidia/cuda:13.4.1
        -image inspect --format {{.Os}}/{{.Architecture}}|{{.Size}} nvidia/cuda:13.4.1
        -tag nvidia/cuda:13.4.1 [WORKSPACE_ID].container.us-west-2.localhost/my-catalog.my-schema.cuda:13.4.1-cudnn-devel-ubuntu26.04
        -push [WORKSPACE_ID].container.us-west-2.localhost/my-catalog.my-schema.cuda:13.4.1-cudnn-devel-ubuntu26.04
        -
        -=== missing image is pulled for AIR; destination defaults to source
        -
        ->>> databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training
        -Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
        -Pushing example/image:v1 as main.training.image:v1 using profile docker-test
        -Configured Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost
        -Updated Docker config: [TEST_TMP_DIR]/home/docker/config.json
        -Installed Docker credential helper: [TEST_TMP_DIR]/home/bin/docker-credential-databricks
        -Pulling source image example/image:v1 for linux/amd64
        -Docker pull progress
        -Source image example/image:v1 is 7.1 GB uncompressed; Databricks Artifact Registry caps images at 30 GB compressed.
        -Tagging image as [WORKSPACE_ID].container.us-west-2.localhost/main.training.image:v1
        -Pushing image to [WORKSPACE_ID].container.us-west-2.localhost/main.training.image:v1
        -Docker push progress
        -Pushed image.
        -Now you can set `environment.unity_catalog_image = main.training.image:v1` in your YAML config and create a workload with `databricks air run -f config.yaml`.
        -
        -=== forced pull and JSON output
        -
        ->>> databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training --pull -o json
        -Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
        -Pushing example/image:v1 as main.training.image:v1 using profile docker-test
        -Configured Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost
        -Updated Docker config: [TEST_TMP_DIR]/home/docker/config.json
        -Installed Docker credential helper: [TEST_TMP_DIR]/home/bin/docker-credential-databricks
        -Pulling source image example/image:v1 for linux/amd64
        -Docker pull progress
        -Source image example/image:v1 is 7.1 GB uncompressed; Databricks Artifact Registry caps images at 30 GB compressed.
        -Tagging image as [WORKSPACE_ID].container.us-west-2.localhost/main.training.image:v1
        -Pushing image to [WORKSPACE_ID].container.us-west-2.localhost/main.training.image:v1
        -Docker push progress
        -Pushed image.
        -
        ->>> jq -e {v, data} [TEST_TMP_DIR]/home/result.json
        -{
        -  "v": 1,
        -  "data": {
        -    "unity_catalog_image": "main.training.image:v1",
        -    "registry_image": "[WORKSPACE_ID].container.us-west-2.localhost/main.training.image:v1"
        -  }
        -}
        -
        -=== daemon failure must not pull
        -
        ->>> musterr databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training
        -Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
        -Pushing example/image:v1 as main.training.image:v1 using profile docker-test
        -Configured Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost
        -Updated Docker config: [TEST_TMP_DIR]/home/docker/config.json
        -Installed Docker credential helper: [TEST_TMP_DIR]/home/bin/docker-credential-databricks
        -Cannot connect to the Docker daemon
        -Error: check local source image example/image:v1 (ensure Docker is running and accessible): exit status 1
        -
        -=== inspect failure must not pull
        -
        ->>> musterr databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training
        -Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
        -Pushing example/image:v1 as main.training.image:v1 using profile docker-test
        -Configured Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost
        -Updated Docker config: [TEST_TMP_DIR]/home/docker/config.json
        -Installed Docker credential helper: [TEST_TMP_DIR]/home/bin/docker-credential-databricks
        -Using local source image example/image:v1
        -Docker inspect failed
        -Error: inspect source image example/image:v1: exit status 1
        -
        -=== incompatible local image must not be pushed
        -
        ->>> musterr databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training
        -Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
        -Pushing example/image:v1 as main.training.image:v1 using profile docker-test
        -Configured Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost
        -Updated Docker config: [TEST_TMP_DIR]/home/docker/config.json
        -Installed Docker credential helper: [TEST_TMP_DIR]/home/bin/docker-credential-databricks
        -Using local source image example/image:v1
        -Error: source image example/image:v1 has platform "linux/arm64"; AIR requires linux/amd64: use --pull to fetch a compatible image, or rebuild the source for linux/amd64
        -
        -=== incompatible pulled image must not be pushed
        -
        ->>> musterr databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training --pull
        -Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
        -Pushing example/image:v1 as main.training.image:v1 using profile docker-test
        -Configured Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost
        -Updated Docker config: [TEST_TMP_DIR]/home/docker/config.json
        -Installed Docker credential helper: [TEST_TMP_DIR]/home/bin/docker-credential-databricks
        -Pulling source image example/image:v1 for linux/amd64
        -Docker pull progress
        -Error: source image example/image:v1 has platform "linux/arm64"; AIR requires linux/amd64: use --pull to fetch a compatible image, or rebuild the source for linux/amd64
        -
        -=== oversized image warns but is pushed
        -
        ->>> databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training
        -Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
        -Pushing example/image:v1 as main.training.image:v1 using profile docker-test
        -Configured Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost
        -Updated Docker config: [TEST_TMP_DIR]/home/docker/config.json
        -Installed Docker credential helper: [TEST_TMP_DIR]/home/bin/docker-credential-databricks
        -Using local source image example/image:v1
        -Source image example/image:v1 is 35.3 GB uncompressed; Databricks Artifact Registry caps images at 30 GB compressed.
        -Warning: source image example/image:v1 exceeds 30 GB uncompressed, so Databricks Artifact Registry may reject the push: start from a Databricks base image (databricksruntime/air) and trim layers such as model weights and build caches.
        -Tagging image as [WORKSPACE_ID].container.us-west-2.localhost/main.training.image:v1
        -Pushing image to [WORKSPACE_ID].container.us-west-2.localhost/main.training.image:v1
        -Docker push progress
        -Pushed image.
        -Now you can set `environment.unity_catalog_image = main.training.image:v1` in your YAML config and create a workload with `databricks air run -f config.yaml`.
        -
        -=== pull failure must not be pushed
        -
        ->>> musterr databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training --pull
        -Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
        -Pushing example/image:v1 as main.training.image:v1 using profile docker-test
        -Configured Docker credential helper for [WORKSPACE_ID].container.us-west-2.localhost
        -Updated Docker config: [TEST_TMP_DIR]/home/docker/config.json
        -Installed Docker credential helper: [TEST_TMP_DIR]/home/bin/docker-credential-databricks
        -Pulling source image example/image:v1 for linux/amd64
        -Docker pull failed
        -Error: pull source image example/image:v1: exit status 1
        -
        -=== invalid destination and removed flags
        -
        ->>> musterr databricks air images push -p docker-test -s example/image:v1 --catalog main --schema training --artifact invalid.name
        -Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
        -Error: invalid artifact name "invalid.name": use lowercase letters and digits separated by one or two underscores or one or more hyphens
        -
        ->>> musterr databricks air images push -p docker-test -s example/image:v1 --catalog main --schema team_-training
        -Warning: This feature is in Preview. APIs may change, and the workspace must enable the Preview features.
        -Error: invalid schema "team_-training": use lowercase letters and digits separated by one or two underscores or one or more hyphens
        -
        ->>> musterr databricks air images push --region us-west-2
        -Error: unknown flag: --region
        -
        -Usage:
        -  databricks air images push [flags]
        -
        -Flags:
        -      --artifact string     Destination ARTIFACT[:TAG]; defaults to the source image name and tag (latest if untagged); maximum 255 characters for catalog.schema.artifact and 128 for the tag
        -      --catalog string      Destination Unity Catalog catalog; use lowercase letters and digits separated by one or two underscores or one or more hyphens
        -  -h, --help[=true|false]   help for push
        -      --pull[=true|false]   Pull the source image even when it is already available locally
        -      --schema string       Destination Unity Catalog schema; use lowercase letters and digits separated by one or two underscores or one or more hyphens
        -  -s, --source string       Source container image to push (NAME[:TAG] or NAME@DIGEST)
        -
        -Global Flags:
        -      --debug[=true|false]   enable debug logging
        -  -o, --output type          output type: text or json (default text)
        -  -p, --profile string       ~/.databrickscfg profile
        -  -t, --target string        bundle target to use (if applicable)
        -
        -
        ->>> musterr databricks air images push --image artifact
        -Error: unknown flag: --image
        -
        -Usage:
        -  databricks air images push [flags]
        -
        -Flags:
        -      --artifact string     Destination ARTIFACT[:TAG]; defaults to the source image name and tag (latest if untagged); maximum 255 characters for catalog.schema.artifact and 128 for the tag
        -      --catalog string      Destination Unity Catalog catalog; use lowercase letters and digits separated by one or two underscores or one or more hyphens
        -  -h, --help[=true|false]   help for push
        -      --pull[=true|false]   Pull the source image even when it is already available locally
        -      --schema string       Destination Unity Catalog schema; use lowercase letters and digits separated by one or two underscores or one or more hyphens
        -  -s, --source string       Source container image to push (NAME[:TAG] or NAME@DIGEST)
        -
        -Global Flags:
        -      --debug[=true|false]   enable debug logging
... (65 more lines truncated, raise DATABRICKS_CLI_TEST_MAX_LOG to see more)

latest (v1.27.0): same output as main (a4388d7)

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at 5e75898. Each finding was verified against the code.

Not ready yet: 1 blocking issue(s), tagged [blocking].

This PR adds an image-size info line and an oversized-image warning to databricks air images push. The size check is well-tested and the warning path (oversized image still pushes) works correctly. However, the parse-failure path hard-aborts the push for what is a purely advisory feature, introducing a new failure mode that the base code doesn't have. A second, minor concern is that the size inspect spawns a redundant subprocess on the same image.

Comment thread cmd/air/image.go Resolved
Comment thread cmd/air/image.go Resolved
Fold {{.Size}} into the platform inspect so no second docker process is
spawned, and warn instead of failing when the size cannot be parsed, since
the size check never gates the push.

Co-authored-by: Isaac <no-reply@databricks.com>
@vinchenzo-db

Copy link
Copy Markdown
Contributor Author

/review

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at a5f8163. Each finding was verified against the code.

No blocking issues found.

This PR adds image-size reporting and a warning to databricks air images push, reading the size from the existing docker image inspect call and printing it before pushing. When the uncompressed size exceeds 20 GB, an advisory warning is shown but the push proceeds. The implementation is clean: edge cases (missing | separator, unparseable size) degrade gracefully into warnings, the format string uses values that cannot contain the delimiter, and tests cover at-limit, over-limit, and unreadable-size scenarios. No concrete defects were found.

@eng-dev-ecosystem-bot

eng-dev-ecosystem-bot commented Nov 21, 2026 •

Copy link
Copy Markdown
Collaborator

✅ Integration test report

Commit: 47a3ded

Run: 44878379480

Env 💚​RECOVERED ✅​pass 🙈​skip Time
💚​ aws linux-2core-8gb 1 272 20 3:55
💚​ aws-windows-latest-4core-16gb 1 274 18 4:04
💚​ azure linux-2core-8gb 1 271 20 3:28
💚​ azure-windows-latest-4core-16gb 1 273 18 4:07
💚​ gcp linux-2core-8gb 1 272 20 3:53
💚​ gcp-windows-latest-4core-16gb 1 274 18 3:10
Test Name aws linux-2core-8gb aws-windows-latest-4core-16gb azure linux-2core-8gb azure-windows-latest-4core-16gb gcp linux-2core-8gb gcp-windows-latest-4core-16gb
💚​ TestAccept 💚​R 💚​R 💚​R 💚​R 💚​R 💚​R
Top 4 slowest tests (at least 2 minutes):
duration env testname
4:05 azure-windows-latest-4core-16gb TestAccept
4:03 aws-windows-latest-4core-16gb TestAccept
3:09 gcp-windows-latest-4core-16gb TestAccept
2:00 gcp linux-2core-8gb TestAccept

@maggiewang-db maggiewang-db left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We have changed the limit: AI Runtime no longer has an image size limit. The only limit now is Databricks Artifact Registry (DAR), which caps images at 30 GB. The 20 GB check and warning in this PR are out of date, so I'm requesting changes to block merging until that's resolved.

AI Runtime no longer limits image size; Databricks Artifact Registry caps
images at 30 GB. Update the threshold, messages, tests, golden output and
changelog fragment accordingly.

Co-authored-by: Isaac <no-reply@databricks.com>
@vinchenzo-db vinchenzo-db changed the title [AIR] CNXT-2687: print the source image size and warn about the 20 GB limit before pushing [AIR] CNXT-2687: print the source image size and warn about the 30 GB registry limit before pushing Nov 21, 2026
@vinchenzo-db

Copy link
Copy Markdown
Contributor Author

/review

@vinchenzo-db

Copy link
Copy Markdown
Contributor Author

Thanks, updated to the 30 GB DAR limit and reworded; AI Runtime's own limit is gone from the PR.

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at cd98a4f. Each finding was verified against the code.

No blocking issues found. 1 non-blocking comment(s) below.

The PR adds a pre-push size check to databricks air images push, printing the uncompressed image size and warning when it exceeds 30 GB. The logic is clean: the inspect format is extended without a second Docker process, parsing failures are advisory, and the push always proceeds. Tests cover at-limit, over-limit, and unparsable cases plus an acceptance test for the oversized path. One minor user-facing text issue noted below.

Comment thread cmd/air/image.go Resolved
Co-authored-by: Isaac <no-reply@databricks.com>
@vinchenzo-db

Copy link
Copy Markdown
Contributor Author

/review

@review-oss-bot review-oss-bot Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review at 47a3ded. Each finding was verified against the code.

No blocking issues found.

This PR adds image-size reporting and a 30 GB advisory warning to databricks air images push, piggy-backing on the existing docker image inspect call. The new code parses the size, prints it, and warns (without blocking) when the uncompressed size exceeds the registry's compressed-layer limit. The implementation is clean: the size/platform parsing handles missing and unparsable values gracefully, tests cover the boundary, over-limit, and unreadable cases, and the acceptance test gains an oversized mode with matching golden output. I found no correctness defects.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIR Databricks AI Runtime CLI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants