Skip to content

Fix duplicate file checks after commit retry - #4446

Open
GayathriSrividya wants to merge 2 commits into
apache:mainfrom
GayathriSrividya:fix/add-files-duplicate-retry-check
Open

GayathriSrividya wants to merge 2 commits into
apache:mainfrom
GayathriSrividya:fix/add-files-duplicate-retry-check

Conversation

@GayathriSrividya

Copy link
Copy Markdown
Contributor

Closes #4445

Rationale for this change

add_files(check_duplicate_files=True) checked duplicate paths only before the first commit attempt. During a conflict retry, the snapshot producer rebuilt the append using refreshed metadata without repeating the duplicate-path validation, allowing concurrent writers to reference the same file twice.

This change registers retry-time validation on snapshot producers and reruns it after refreshing metadata. Duplicate paths now raise ValueError before the retried snapshot is rebuilt.

Are these changes tested?

  • ruff check pyiceberg/table/__init__.py pyiceberg/table/update/snapshot.py tests/table/test_commit_retry.py — passed
  • ruff format --check pyiceberg/table/__init__.py pyiceberg/table/update/snapshot.py tests/table/test_commit_retry.py — passed
  • python -m compileall -q pyiceberg/table/__init__.py pyiceberg/table/update/snapshot.py tests/table/test_commit_retry.py — passed
  • git diff --check — passed
  • The targeted pytest regression test could not complete because PyArrow imports stalled in the local environment.

Are there any user-facing changes?

Yes. Concurrent add_files calls can no longer add the same file twice after a commit retry. Duplicate files are rejected with the existing duplicate-file ValueError.

Revalidate add_files duplicate paths against the refreshed branch snapshot before rebuilding a retried snapshot.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great catch @GayathriSrividya 🙌 And thanks for working on this. I left one suggestion. LMKWYT

Comment thread pyiceberg/table/__init__.py Outdated
Comment on lines 1298 to 1299
producer._validate_retry()
producer._validate_concurrency()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two methods look similar, could we merge the _validate_retry() into _validate_concurrency()

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion @Fokko , I merged the retry validation into _validate_concurrency() while preserving fast append’s behavior. The commit-retry tests pass locally.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

add_files(check_duplicate_files=True) can commit duplicate paths after a conflict retry

2 participants