Repository navigation
Conversation
The client only allowed GPU_1xA10 and GPU_8xH100, so newer AI Runtime GPU types such as GPU_1xRTX6000 were rejected before the job was submitted even though the server side never looks at the accelerator. Validate the GPU_<count>x<model> shape instead and let the workspace decide which types it offers. Co-authored-by: Isaac <no-reply@databricks.com>
Co-authored-by: Isaac <no-reply@databricks.com>
|
Thanks for the contribution! This pull request was opened from a fork. If you're a member of the Databricks organization, please request access to the If you're an external contributor, no action is needed; a maintainer will review your pull request. |
Export the accelerator types `databricks air` accepts as KnownGPUTypes and reuse them in the ssh client instead of a shape regex, so the two commands agree on the GPU types they offer. Co-authored-by: Isaac <no-reply@databricks.com>
ssh connect --acceleratorssh connect --accelerator
| // KnownGPUTypes lists the accelerator types this CLI accepts, in the order they are shown | ||
| // to users. `ssh connect --accelerator` validates against the same list, so the two entry | ||
| // points to AI Runtime GPUs agree on the types they offer. | ||
| func KnownGPUTypes() []string { |
There was a problem hiding this comment.
can we call AICM's ListComputeOptions to get the list of gpu types available to the user in their workspace? AIR CLI does this already.
There was a problem hiding this comment.
When Jobs provisions the SSH session, AICM’s CreateSession already checks whether the workspace can use the requested GPU. Do we really need the check in CLI using one additional RPC call?
There was a problem hiding this comment.
I traced the launch path to check what the backend already does before adding an RPC here. ssh connect submits a Jobs run with the accelerator on the task, and Jobs provisions it through AI Compute Manager's CreateSession, the same handler the notebook attach path uses. That handler validates the GPU type against the configured types, rejects types the workspace is not allowed to use (<type> is not available in your workspace.), and applies the same B300 gate that ListComputeOptions applies. Jobs does not retry or swallow those errors: the run terminates and the CLI prints the message from the task's termination details (see the live example in the thread below on the environment-version check).
So a ListComputeOptions call in the CLI would duplicate the server check, and would still tie new GPU types to CLI behavior. I updated the PR the other way: the CLI no longer carries a GPU allowlist at all. It only checks that the value is a safe GPU_... identifier, since it is interpolated into the SSH ProxyCommand, and relies on the backend for availability. For discovery, the flag help points at databricks air run -h config.compute.accelerator_type, which already lists the workspace's compute options.
| // which ones a workspace can use comes from the server, not this CLI. | ||
| func parseGPUType(value string) (gpuType, error) { | ||
| switch gpuType(value) { | ||
| case gpuType1xA10, gpuType8xH100, gpuType1xH100, gpuType8xB300, gpuType1xRTX6000: |
There was a problem hiding this comment.
@caroline-db I thought you already removed this list from the CLI client code in favor of the server-side GPU type list. Is this one still used?
There was a problem hiding this comment.
This PR no longer touches cmd/air; I dropped the shared KnownGPUTypes export, so air is unchanged here. For context, parseGPUType in air is still load-bearing today: air run validates config.compute.accelerator_type against it before submission and uses the per-type GPU count for container rank validation and log download, while the compute-options endpoint feeds the help text and the air list filter. Moving air fully onto the server list is a separate change, best left to caroline-db.
| if o.Accelerator != "" && o.Accelerator != "GPU_1xA10" && o.Accelerator != "GPU_8xH100" { | ||
| return fmt.Errorf("invalid accelerator value: %q, expected %q or %q", o.Accelerator, "GPU_1xA10", "GPU_8xH100") | ||
| // The same GPU types `databricks air` accepts; the workspace still decides which it offers. | ||
| if o.Accelerator != "" && !slices.Contains(aircmd.KnownGPUTypes(), o.Accelerator) { |
There was a problem hiding this comment.
The validation is more nuanced: Not all environment versions work with all GPU types. For example, B300 requires v6+. Can you change the validation here to check each environment-version × accelerator combination?
Even better: move the validation to server side (AI Training) and then we can add more GPU types any time without asking the users to upgrade CLI.
There was a problem hiding this comment.
The backend enforces this on the same launch path, so I would rather not duplicate the matrix in the CLI. AI Compute Manager's CreateSession validates the requested image version against the accelerator (B300 requires version 6 or later), and the CLI surfaces the failure. Verified live in a staging workspace with this change:
$ databricks ssh connect --name probe --accelerator=GPU_8xB300 --environment-version=5
...
Job submitted successfully with run ID: <run id>
Provisioning GPU_8xB300 compute. This can take several minutes and may take longer when capacity is constrained.
Error: failed to ensure that ssh server is running: failed to submit and start ssh server job: ssh server bootstrap job failed:
Unexpected user error while preparing the cluster for the job. Cause: BAD_REQUEST: GPU_8xB300 requires client image version >= 6, got: 5
The PR now does what you suggested as "even better": the CLI only checks that the value is a safe identifier for the ProxyCommand, and leaves GPU type, workspace availability and environment-version compatibility to the server, so new GPU types or version requirements need no CLI release.
Reviewers asked us not to hardcode GPU types or a GPU-by-environment matrix. The backend already validates both on the job submission path, and failures are surfaced from the run's termination details. Co-authored-by: Isaac <no-reply@databricks.com>
|
An authorized user can trigger integration tests manually by following the instructions below: Trigger: Inputs:
Checks will be approved automatically on success. |
Changes
databricks ssh connect --acceleratorno longer validates the value against a GPU allowlist built into the CLI. It only checks that the value is a safeGPU_...identifier (letters, numbers, dashes and underscores), because the value is interpolated into the OpenSSH ProxyCommand and the generated connection name. Which GPU types a workspace offers, and which environment versions they support, is validated by the backend when the SSH server job is submitted. The flag help points atdatabricks air run -h config.compute.accelerator_type, which already lists the workspace's compute options.cmd/airis unchanged by this PR.Why
The ssh client had its own allowlist of two GPU types, so newer AI Runtime GPUs such as RTX6000 were rejected before the SSH server job was submitted. Reviewers asked not to replace it with another hardcoded list or a GPU-by-environment-version matrix in the CLI. The backend already enforces both on the same launch path: Jobs provisions the session through AI Compute Manager's CreateSession, which validates the GPU type, whether the workspace may use it, and the environment-version requirement (B300 needs version 6 or later), and the failure is surfaced by the CLI from the task's termination details. Keeping the CLI to a shape check means new GPU types or version requirements need no CLI release.
Tests
go test ./experimental/ssh/... ./cmd/air/...passes.Validatetests coverGPU_1xRTX6000, a well-formed but unknownGPU_2xA10(now accepted; the server decides),CPU_1x(rejected), and two shell-metacharacter values (rejected).TestBuildSSHServerSubmitRungains a pass-through case for an unknown accelerator.gofmt -l,go vetand the repository-pinnedgolangci-lintreport no issues on the changed packages.TestAccept/ssh/connect-serverless-gputimes out in my environment after the fake server reportsConnected!; it times out identically on the unchanged previous head, so it is environmental and unrelated to this change.--releases-dir:databricks ssh connect --name probe --accelerator=GPU_8xB300 --environment-version=5submits the job and then fails with the backend message surfaced by the CLI:start_ssh_servertask terminates withINVALID_CLUSTER_REQUESTand the same message.databricks ssh connect --accelerator=GPU_1xRTX6000 --environment-version=6started the SSH server job on a 1xRTX6000 serverless GPU in a staging workspace and the SSH handshake completed. The unpatched CLI fails the same command withinvalid accelerator value.validate-generatedandtask test-exp-sshfail in their setup step because fork PRs get no OIDC token for the JFrog proxy, before they reach this change.This pull request and its description were written by Isaac.