Skip to content

Fix duplicate file checks after commit retry - #4446

Open
GayathriSrividya wants to merge 3 commits into
apache:mainfrom
GayathriSrividya:fix/add-files-duplicate-retry-check
Open

GayathriSrividya wants to merge 3 commits into
apache:mainfrom
GayathriSrividya:fix/add-files-duplicate-retry-check

Conversation

@GayathriSrividya

Copy link
Copy Markdown
Contributor

Closes #4445

Rationale for this change

add_files(check_duplicate_files=True) checked duplicate paths only before the first commit attempt. During a conflict retry, the snapshot producer rebuilt the append using refreshed metadata without repeating the duplicate-path validation, allowing concurrent writers to reference the same file twice.

This change registers retry-time validation on snapshot producers and reruns it after refreshing metadata. Duplicate paths now raise ValueError before the retried snapshot is rebuilt.

Are these changes tested?

  • ruff check pyiceberg/table/__init__.py pyiceberg/table/update/snapshot.py tests/table/test_commit_retry.py — passed
  • ruff format --check pyiceberg/table/__init__.py pyiceberg/table/update/snapshot.py tests/table/test_commit_retry.py — passed
  • python -m compileall -q pyiceberg/table/__init__.py pyiceberg/table/update/snapshot.py tests/table/test_commit_retry.py — passed
  • git diff --check — passed
  • The targeted pytest regression test could not complete because PyArrow imports stalled in the local environment.

Are there any user-facing changes?

Yes. Concurrent add_files calls can no longer add the same file twice after a commit retry. Duplicate files are rejected with the existing duplicate-file ValueError.

Revalidate add_files duplicate paths against the refreshed branch snapshot before rebuilding a retried snapshot.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great catch @GayathriSrividya 🙌 And thanks for working on this. I left one suggestion. LMKWYT

Comment thread pyiceberg/table/__init__.py Outdated
Comment on lines 1298 to 1299
producer._validate_retry()
producer._validate_concurrency()

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These two methods look similar, could we merge the _validate_retry() into _validate_concurrency()

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion @Fokko , I merged the retry validation into _validate_concurrency() while preserving fast append’s behavior. The commit-retry tests pass locally.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Appreciate the quick follow-up 🙌 Let me check!

Comment thread pyiceberg/table/__init__.py Outdated
break
self._rebuild_snapshot_updates()
except (CommitFailedException, ValidationException):
except (CommitFailedException, ValidationException, ValueError):

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more comment! I think this is a risky change. The ValueError can be general catch-all. Let's say that the commit succeeded, but there is an issue with the response. In this situation all the files will be cleaned up. Can we avoid this change? For example, when it find a duplicate file, that we raise a ValidationException instead?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for catching this, @Fokko . You’re right that catching ValueError broadly could trigger cleanup when the commit outcome isn’t known to be safe. I’ve updated retry-time duplicate detection to raise ValidationException and removed ValueError from the cleanup handler, while preserving ValueError for the initial duplicate check. Sorry for overlooking that cleanup-safety implication in my earlier change. The focused commit-retry tests pass (174 passed).

Raise ValidationException when a duplicate is discovered during a commit retry so cleanup only runs for validation failures known to be safe. Keep ValueError for the initial duplicate check.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

add_files(check_duplicate_files=True) can commit duplicate paths after a conflict retry

2 participants