Skip to content

Accept the AI Runtime GPU types in ssh connect --accelerator - #7833

Open
apeforest wants to merge 4 commits into
databricks:mainfrom
apeforest:ssh-connect-accept-any-gpu-accelerator
Open

apeforest wants to merge 4 commits into
databricks:mainfrom
apeforest:ssh-connect-accept-any-gpu-accelerator

Conversation

@apeforest

@apeforest apeforest commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Changes

databricks ssh connect --accelerator no longer validates the value against a GPU allowlist built into the CLI. It only checks that the value is a safe GPU_... identifier (letters, numbers, dashes and underscores), because the value is interpolated into the OpenSSH ProxyCommand and the generated connection name. Which GPU types a workspace offers, and which environment versions they support, is validated by the backend when the SSH server job is submitted. The flag help points at databricks air run -h config.compute.accelerator_type, which already lists the workspace's compute options. cmd/air is unchanged by this PR.

Why

The ssh client had its own allowlist of two GPU types, so newer AI Runtime GPUs such as RTX6000 were rejected before the SSH server job was submitted. Reviewers asked not to replace it with another hardcoded list or a GPU-by-environment-version matrix in the CLI. The backend already enforces both on the same launch path: Jobs provisions the session through AI Compute Manager's CreateSession, which validates the GPU type, whether the workspace may use it, and the environment-version requirement (B300 needs version 6 or later), and the failure is surfaced by the CLI from the task's termination details. Keeping the CLI to a shape check means new GPU types or version requirements need no CLI release.

Tests

  • Unit: go test ./experimental/ssh/... ./cmd/air/... passes. Validate tests cover GPU_1xRTX6000, a well-formed but unknown GPU_2xA10 (now accepted; the server decides), CPU_1x (rejected), and two shell-metacharacter values (rejected). TestBuildSSHServerSubmitRun gains a pass-through case for an unknown accelerator.
  • Lint: gofmt -l, go vet and the repository-pinned golangci-lint report no issues on the changed packages.
  • The local acceptance test TestAccept/ssh/connect-serverless-gpu times out in my environment after the fake server reports Connected!; it times out identically on the unchanged previous head, so it is environmental and unrelated to this change.
  • Manual, staging workspace, with this change built and uploaded via --releases-dir: databricks ssh connect --name probe --accelerator=GPU_8xB300 --environment-version=5 submits the job and then fails with the backend message surfaced by the CLI:
    Job submitted successfully with run ID: <run id>
    Provisioning GPU_8xB300 compute. This can take several minutes and may take longer when capacity is constrained.
    Error: failed to ensure that ssh server is running: failed to submit and start ssh server job: ssh server bootstrap job failed:
      Unexpected user error while preparing the cluster for the job. Cause: BAD_REQUEST: GPU_8xB300 requires client image version >= 6, got: 5
    
    The start_ssh_server task terminates with INVALID_CLUSTER_REQUEST and the same message.
  • Earlier manual test (unchanged): databricks ssh connect --accelerator=GPU_1xRTX6000 --environment-version=6 started the SSH server job on a 1xRTX6000 serverless GPU in a staging workspace and the SSH handshake completed. The unpatched CLI fails the same command with invalid accelerator value.
  • CI note: validate-generated and task test-exp-ssh fail in their setup step because fork PRs get no OIDC token for the JFrog proxy, before they reach this change.

This pull request and its description were written by Isaac.

The client only allowed GPU_1xA10 and GPU_8xH100, so newer AI Runtime GPU
types such as GPU_1xRTX6000 were rejected before the job was submitted even
though the server side never looks at the accelerator. Validate the
GPU_<count>x<model> shape instead and let the workspace decide which types
it offers.

Co-authored-by: Isaac <no-reply@databricks.com>
@apeforest
apeforest requested review from a team as code owners November 20, 2026 00:20
Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions

Copy link
Copy Markdown
Contributor

Thanks for the contribution! This pull request was opened from a fork.

If you're a member of the Databricks organization, please request access to the cli-contrib team and open your pull request from a branch in this repository instead of a fork — fork PRs don't run integration tests automatically and have reduced CI access.

If you're an external contributor, no action is needed; a maintainer will review your pull request.

Export the accelerator types `databricks air` accepts as KnownGPUTypes and
reuse them in the ssh client instead of a shape regex, so the two commands
agree on the GPU types they offer.

Co-authored-by: Isaac <no-reply@databricks.com>
@apeforest
apeforest requested a review from a team as a code owner November 20, 2026 00:32
@github-actions github-actions Bot added the AIR Databricks AI Runtime CLI label Nov 20, 2026
@apeforest apeforest changed the title Accept any serverless GPU type in ssh connect --accelerator Accept the AI Runtime GPU types in ssh connect --accelerator Nov 20, 2026
Comment thread cmd/air/compute.go Outdated
// KnownGPUTypes lists the accelerator types this CLI accepts, in the order they are shown
// to users. `ssh connect --accelerator` validates against the same list, so the two entry
// points to AI Runtime GPUs agree on the types they offer.
func KnownGPUTypes() []string {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

can we call AICM's ListComputeOptions to get the list of gpu types available to the user in their workspace? AIR CLI does this already.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When Jobs provisions the SSH session, AICM’s CreateSession already checks whether the workspace can use the requested GPU. Do we really need the check in CLI using one additional RPC call?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I traced the launch path to check what the backend already does before adding an RPC here. ssh connect submits a Jobs run with the accelerator on the task, and Jobs provisions it through AI Compute Manager's CreateSession, the same handler the notebook attach path uses. That handler validates the GPU type against the configured types, rejects types the workspace is not allowed to use (<type> is not available in your workspace.), and applies the same B300 gate that ListComputeOptions applies. Jobs does not retry or swallow those errors: the run terminates and the CLI prints the message from the task's termination details (see the live example in the thread below on the environment-version check).

So a ListComputeOptions call in the CLI would duplicate the server check, and would still tie new GPU types to CLI behavior. I updated the PR the other way: the CLI no longer carries a GPU allowlist at all. It only checks that the value is a safe GPU_... identifier, since it is interpolated into the SSH ProxyCommand, and relies on the backend for availability. For discovery, the flag help points at databricks air run -h config.compute.accelerator_type, which already lists the workspace's compute options.

Comment thread cmd/air/compute.go
// which ones a workspace can use comes from the server, not this CLI.
func parseGPUType(value string) (gpuType, error) {
switch gpuType(value) {
case gpuType1xA10, gpuType8xH100, gpuType1xH100, gpuType8xB300, gpuType1xRTX6000:

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@caroline-db I thought you already removed this list from the CLI client code in favor of the server-side GPU type list. Is this one still used?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR no longer touches cmd/air; I dropped the shared KnownGPUTypes export, so air is unchanged here. For context, parseGPUType in air is still load-bearing today: air run validates config.compute.accelerator_type against it before submission and uses the per-type GPU count for container rank validation and log download, while the compute-options endpoint feeds the help text and the air list filter. Moving air fully onto the server list is a separate change, best left to caroline-db.

if o.Accelerator != "" && o.Accelerator != "GPU_1xA10" && o.Accelerator != "GPU_8xH100" {
return fmt.Errorf("invalid accelerator value: %q, expected %q or %q", o.Accelerator, "GPU_1xA10", "GPU_8xH100")
// The same GPU types `databricks air` accepts; the workspace still decides which it offers.
if o.Accelerator != "" && !slices.Contains(aircmd.KnownGPUTypes(), o.Accelerator) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The validation is more nuanced: Not all environment versions work with all GPU types. For example, B300 requires v6+. Can you change the validation here to check each environment-version × accelerator combination?

Even better: move the validation to server side (AI Training) and then we can add more GPU types any time without asking the users to upgrade CLI.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The backend enforces this on the same launch path, so I would rather not duplicate the matrix in the CLI. AI Compute Manager's CreateSession validates the requested image version against the accelerator (B300 requires version 6 or later), and the CLI surfaces the failure. Verified live in a staging workspace with this change:

$ databricks ssh connect --name probe --accelerator=GPU_8xB300 --environment-version=5
...
Job submitted successfully with run ID: <run id>
Provisioning GPU_8xB300 compute. This can take several minutes and may take longer when capacity is constrained.
Error: failed to ensure that ssh server is running: failed to submit and start ssh server job: ssh server bootstrap job failed:
  Unexpected user error while preparing the cluster for the job. Cause: BAD_REQUEST: GPU_8xB300 requires client image version >= 6, got: 5

The PR now does what you suggested as "even better": the CLI only checks that the value is a safe identifier for the ProxyCommand, and leaves GPU type, workspace availability and environment-version compatibility to the server, so new GPU types or version requirements need no CLI release.

Reviewers asked us not to hardcode GPU types or a GPU-by-environment matrix.
The backend already validates both on the job submission path, and failures
are surfaced from the run's termination details.

Co-authored-by: Isaac <no-reply@databricks.com>
@github-actions

Copy link
Copy Markdown
Contributor

An authorized user can trigger integration tests manually by following the instructions below:

Trigger:
go/deco-tests-run/cli

Inputs:

  • PR number: 7833
  • Commit SHA: 5834febf1841e3cdc41a3d687febaf9da733498b

Checks will be approved automatically on success.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

AIR Databricks AI Runtime CLI

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants