Skip to content

fix(storage): keep object metadata when initial MRD open is redirected with a read handle - #19135

Open
yuxin00j wants to merge 1 commit into
googleapis:mainfrom
yuxin00j:storage-mrd-redirect-read-handle
Open

yuxin00j wants to merge 1 commit into
googleapis:mainfrom
yuxin00j:storage-mrd-redirect-read-handle

Conversation

@yuxin00j

Copy link
Copy Markdown
Contributor

Summary

When AsyncMultiRangeDownloader.open() is called without a read_handle and receives a BidiReadObjectRedirectedError containing both routing_token and read_handle, _on_open_error() previously adopted the redirect's read_handle. Because the server omits BidiReadObjectResponse.metadata when a stream is opened with a read_handle, create_mrd() returned with persisted_size=None, is_finalized=False, full_obj_server_crc32c=None, and generation=None. _on_open_error() now retries with routing_token alone unless the downloader already has a read_handle or the redirect has no routing_token.

Fixes #19134

Changes

  • AsyncMultiRangeDownloader._on_open_error() (packages/google-cloud-storage/google/cloud/storage/asyncio/async_multi_range_downloader.py): only store redirect.read_handle when self.read_handle is already set or routing_token is absent; document the behavior.
  • Add unit tests in packages/google-cloud-storage/tests/unit/asyncio/test_async_multi_range_downloader.py:
    • test_open_redirect_with_read_handle_keeps_object_metadata (initial open without handle, redirect with token + handle -> retries with token only and populates persisted_size, is_finalized, full_obj_server_crc32c, generation, and server-returned read_handle).
    • test_open_redirect_adopts_read_handle (parametrized over caller-supplied read_handle and handle-only redirect without routing_token -> keeps adopting redirect.read_handle).

Testing

  • Unit tests: pytest tests/unit/asyncio/test_async_multi_range_downloader.py -k "test_open_redirect_" fails before the fix (read_handle adopted on retry, persisted_size is None) and passes with it (3 passed).
  • Full tests/unit/asyncio/ suite: 408 passed, 18 skipped (only the 2 pre-existing local caplog-propagation failures in test_create_mrd_with_generation_number and test_unknown_read_id_is_dropped that fail identically on main).
  • Lint/format: ruff check, ruff format --check, and flake8 pass on both changed files.
  • Live check against a zonal bucket with an unfinalized appendable object (36 bytes): injecting a redirect carrying a real routing token and real read handle on the first open attempt produced persisted_size=None, generation=None on main and persisted_size=36 with the object's generation after this fix.

…d with a read handle

When AsyncMultiRangeDownloader.open() is called without a read_handle and receives a BidiReadObjectRedirectedError containing both routing_token and read_handle, _on_open_error() previously adopted redirect.read_handle. Because the server omits BidiReadObjectResponse.metadata when a stream is opened with a read_handle, create_mrd() returned with persisted_size=None, is_finalized=False, full_obj_server_crc32c=None, and generation=None.

Only adopt redirect.read_handle in _on_open_error() when self.read_handle is already set or routing_token is absent; otherwise retry with routing_token alone so the initial response includes object metadata and a fresh read_handle.
@yuxin00j
yuxin00j requested review from a team as code owners November 19, 2026 08:20

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request updates the _on_open_error method in AsyncMultiRangeDownloader to conditionally use the redirect's read handle. Specifically, the read handle is only adopted if the downloader already has a read handle or if the redirect lacks a routing token. This ensures that the subsequent retry attempt can retrieve the object metadata (such as persisted_size, is_finalized, and full_obj_server_crc32c) in its first response when a routing token is present. Additionally, comprehensive unit tests have been added to verify this behavior under different redirect scenarios. There are no review comments to address, and the changes look solid.

@yuxin00j

Copy link
Copy Markdown
Contributor Author

/gcbrun

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

storage: AsyncMultiRangeDownloader loses object metadata when initial open is redirected with a read_handle

1 participant