Repository navigation
Complete configured Volume transfers through the native backend - #613
aakash-saravanan-db wants to merge 46 commits into
Conversation
|
Thanks for your contribution! To satisfy the DCO policy in our contributing guide every commit message must include a sign-off message. One or more of your commits is missing this message. You can reword previous commit messages with an interactive rebase ( |
ad49367 to
7b276d7
Compare
7b276d7 to
12876fc
Compare
There was a problem hiding this comment.
Verdict: 1 Medium · 1 Low
Mostly sound, with one medium issue: on the kernel path, the new StagingError wrapping can also catch the statement's own execution errors, not just transfer errors. Nits: there is a stray blank line in the CHANGELOG entry, and the constructor comment in KernelOperationBackend still says cancel() "always routes to the cancellable execution". Follow-up (not caused by this PR): executeStatement waits for the transfer before it returns the operation, so a caller can only interrupt an in-flight transfer by closing the session.
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 High · 2 Medium · 2 Low
Needs changes: 3 medium and 1 low concern. The staging dispatch and error mapping look right, but the PR ships behavior that depends on the unreleased native binding with no capability gate, and it changes socketTimeout: 0 semantics without a test or a CHANGELOG entry.
This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.
| const handle = await this.getFetchHandle(); | ||
| let outcome; | ||
| try { | ||
| outcome = await handle.completeVolumeOperation(allowedPaths, true); // Thrift REMOVE treats missing objects as success. |
There was a problem hiding this comment.
🟠 High — completeStagingOperation calls handle.completeVolumeOperation unconditionally, and assertBindingShape never checks for it. With an older installed binding (platform packages are still pinned to 1.4.0), kernel executeStatement with stagingAllowedLocalPath throws a raw TypeError. Previously it warned and returned results. Add a capability check that gives a clear error, and bump the pins together with this change.
🔀 Found by both reviewers · confirmed by the judge
There was a problem hiding this comment.
Release gate. The kernel ships as 1.5.0 first; this PR then moves every platform package pin and the lockfile to 1.5.0 before merge, so npm installs get a binding with completeVolumeOperation and the zero-timeout handling. No capability check is added: the TypeScript and the binding ship as one pinned set. Keeping this thread open until the pins move.
There was a problem hiding this comment.
Still open at 0947cc8: the platform pins in package.json:95-102 are still 1.4.0, so this blocks merge until they move to 1.5.0.
| - Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608) | ||
| - Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 16 and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open. | ||
| - Reduce result chunking and `fetchAll()` overhead for large query results. | ||
| - Support Volume file transfers on the kernel backend when `stagingAllowedLocalPath` is supplied. Symlink targets must remain within configured allowed roots; in-root symlinks are permitted. |
There was a problem hiding this comment.
🔵 Low — The changelog entry doesn't say this needs a newer native kernel package than the pinned 1.4.0, and it doesn't reference the kernel change. Add the required binding version and the kernel PR.
🔀 Found by Claude Opus 5.5 · confirmed by the judge
| // default — preserving the "effectively no idle timeout" semantics. | ||
| if (typeof socketTimeout === 'number' && socketTimeout > 0) { | ||
| // Preserve zero so native storage inactivity can be disabled like Thrift. | ||
| if (typeof socketTimeout === 'number' && socketTimeout >= 0) { |
There was a problem hiding this comment.
🟡 Medium — socketTimeout: 0 is now forwarded as socketTimeoutMs: 0. Bindings that still treat the value as the reqwest total timeout (the currently pinned ones) would time out immediately. The change has no unit test. Gate it on the coordinated binding release, add tests for 0 and positive values, and add a changelog note.
🔀 Found by Claude Opus 5.5 · confirmed by the judge
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 High · 1 Medium · 2 Low
Mostly sound, with 2 medium concerns. The staging dispatch and error mapping look correct. Merging is still blocked, as the PR description says, on the platform package pin and lockfile updates.
This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.
Other findings
- 🟠 High — Completion now unconditionally requires the new native API, but npm installations still receive the pinned 1.4.0 packages rather than the coordinated release required by this PR; updating KERNEL_REV only changes source builds. Update every platform package pin and lockfile to the compatible release and validate staging through a clean installed driver before merging. (Found by GPT-6.1 Sol · the judge couldn't confirm it)
| // default — preserving the "effectively no idle timeout" semantics. | ||
| if (typeof socketTimeout === 'number' && socketTimeout > 0) { | ||
| // Preserve zero so native storage inactivity can be disabled like Thrift. | ||
| if (typeof socketTimeout === 'number' && socketTimeout >= 0) { |
There was a problem hiding this comment.
🔵 Low — socketTimeout 0 is now forwarded to the binding, and positive values also set the storage idle timeout. There is no unit test for buildKernelHttpOptions covering this, and the CHANGELOG has no entry for it. Add tests for 0 and positive values, and a CHANGELOG note that it needs the new native release.
🔀 Found by Claude Opus 5.5 · confirmed by the judge
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 High · 1 Medium · 2 Low
Mostly sound: 1 medium and 1 low concern. The PR already notes that native pins are pending. Until they land, an older binding has no completeVolumeOperation, so every statement run with stagingAllowedLocalPath would throw a TypeError on the kernel backend instead of the old no-op.
This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 High · 1 Medium · 2 Low
Needs changes: 1 high, 1 medium. With the currently pinned 1.4.0 native binaries, the staging completion call and the forwarded socketTimeout: 0 would break kernel users until the compatible binding pins land in this PR.
| - Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608) | ||
| - Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 16 and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open. | ||
| - Reduce result chunking and `fetchAll()` overhead for large query results. | ||
| - Support Volume file transfers on the kernel backend when `stagingAllowedLocalPath` is supplied. Symlink targets must remain within configured allowed roots; in-root symlinks are permitted. (#613) |
There was a problem hiding this comment.
🔵 Low — The changelog entry doesn't say a published kernel package bump is required. It also doesn't mention that socketTimeout: 0 is now forwarded and that the timeout now bounds storage idle time. Add both notes as separate entries.
🔀 Found by Claude Opus 5.5 · confirmed by the judge
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 High · 1 Medium · 3 Low
Looks reasonable, with 1 medium and 1 low concern. The kernel staging dispatch, error mapping and tests match the Thrift path's behavior. As the PR description says, the native platform pins and locks still have to be bumped before merge.
This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.
| - Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608) | ||
| - Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 16 and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open. | ||
| - Reduce result chunking and `fetchAll()` overhead for large query results. | ||
| - Support Volume file transfers on the kernel backend when `stagingAllowedLocalPath` is supplied. Symlink targets must remain within configured allowed roots; in-root symlinks are permitted. (#613) |
There was a problem hiding this comment.
🔵 Low — The changelog entry doesn't mention that kernel socketTimeout semantics changed. Positive values now also set a storage idle timeout, and 0 is forwarded to disable it. Add that behavior change to the entry.
🔀 Found by Claude Opus 5.5 · confirmed by the judge
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).
Verdict: 2 High · 2 Low
Needs changes — 1 high, 1 medium, 1 low. The delegation logic looks right, but the code calls native APIs and relies on native semantics that the pinned 1.4.0 binaries don't provide.
This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.
| - Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608) | ||
| - Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 16 and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open. | ||
| - Reduce result chunking and `fetchAll()` overhead for large query results. | ||
| - Support Volume file transfers on the kernel backend when `stagingAllowedLocalPath` is supplied. Symlink targets must remain within configured allowed roots; in-root symlinks are permitted. (#613) |
There was a problem hiding this comment.
🔵 Low — The changelog entry omits the required published kernel package bump and the socketTimeout change on kernel, where 0 is now forwarded and storage gets an idle timeout. Add both to the entry.
🔀 Found by Claude Opus 5.5 · confirmed by the judge
Signed-off-by: Aakash Saravanan <aakash.saravanan@databricks.com>
There was a problem hiding this comment.
🔀 Cross-reviewed: two independent reviews (Claude Opus 5.5 · GPT-6.1 Sol), checked by a judge model (Claude Sonnet 5.5).
Verdict: 1 High · 3 Low
Mostly sound, with 1 medium and 2 low concerns. The dispatch, completion and error mapping hold together, but the code assumes native APIs that the pinned 1.4.0 bindings don't ship yet.
This overview is Claude Opus 5.5's initial take; GPT-6.1 Sol added findings it doesn't mention — see the findings below.
| if (opBackend.completeStagingOperation) { | ||
| await this.runBackend(() => opBackend.completeStagingOperation!(allowedLocalPath)); | ||
| return operation; | ||
| } |
There was a problem hiding this comment.
🔵 Low — If native completion rejects, executeStatement throws without closing the already-registered operation, so the kernel statement stays open until the session closes. Best-effort close the operation before rethrowing, and keep the original error.
(Anchored to the nearest changed line — see the description for the exact location.)
🔀 Found by GPT-6.1 Sol · confirmed by the judge
| const handle = await this.getFetchHandle(); | ||
| let outcome; | ||
| try { | ||
| outcome = await handle.completeVolumeOperation(allowedPaths, true); // Thrift REMOVE treats missing objects as success. |
There was a problem hiding this comment.
🟠 High — The code calls completeVolumeOperation with no check that it exists, and package.json still pins the kernel packages at 1.4.0. With those binaries, any kernel statement run with stagingAllowedLocalPath fails with an opaque TypeError. Guard with a typeof check and throw a clear error naming the required version, or bump the pins in the same change.
🔀 Found by both reviewers · confirmed by the judge
| - Kernel backend: a failed status poll while waiting for a statement (a `runAsync: true` statement, or a default execution that outlives the server's inline wait) and a failed result fetch in `hasMoreRows()` now throw the mapped driver error, such as `HiveDriverError: HTTP 500: Internal Server Error` once retries on HTTP 5xx run out, instead of a raw `Error` whose message is the internal `__databricks_error__:{...}` envelope. As in `fetchChunk()`, a failed fetch in `hasMoreRows()` also closes the statement. (#608) | ||
| - Honor `cloudFetchConcurrentDownloads` when opening kernel sessions, with a default window of 16 and explicit chunk-setting overrides. Values other than positive safe integers raise `HiveDriverError` at session open. | ||
| - Reduce result chunking and `fetchAll()` overhead for large query results. | ||
| - Support Volume file transfers on the kernel backend when `stagingAllowedLocalPath` is supplied. Symlink targets must remain within configured allowed roots; in-root symlinks are permitted. (#613) |
There was a problem hiding this comment.
🔵 Low — The changelog entry says Volume transfers are supported but doesn't name the native package version they need or the kernel change. The published 1.4.0 packages lack the binding. Add the required kernel package version, or note that a follow-up bump is needed, and reference the kernel PR.
🔀 Found by Claude Opus 5.5 · confirmed by the judge
| (mapped instanceof ParameterError && !metadata?.httpStatus) || | ||
| metadata?.errorCode === 'VOLUME_STORAGE_HTTP_ERROR' | ||
| ) { | ||
| const staging = new StagingError(mapped.message); |
There was a problem hiding this comment.
🔵 Low — toStagingTransferError and the local-file branch of decodeNapiKernelError each replace the mapped error with a new one. Neither copies sqlState, so the SQLSTATE that mapKernelErrorToJsError attached is lost. Copy sqlState with defineErrorMetadata in both places and test it.
🔀 Found by both reviewers · confirmed by the judge
Feature or gap
Configured staging operations need to finish their Volume transfer through the native backend before executeStatement resolves.
What this PR addresses
Delegate marked file PUT, GET and REMOVE to the shared native core, preserve raw results when staging is omitted, and map staging and local-file errors through the Node API. Native completion declarations and enum exports are generated from the binding source.
Dependencies
Requires the Volume APIs in databricks/databricks-sql-kernel#465 and a compatible coordinated native release. Platform package pins and locks, plus clean installed-driver validation, must be updated before merge.
Kernel version
The compatible kernel release has not been selected. When it is available, update every platform package pin and lockfile in this PR and validate a clean installation before merge.
KERNEL_REVselects source for local and CI builds only.Tests and validation
The focused suites pass: 306 tests. TypeScript build, production ESLint and formatting pass. Native declarations use NAPI-RS CLI 3.8.2.