Skip to content

feat: add table-aware position delete updates - #919

Draft
wirybeaver wants to merge 4 commits into
apache:mainfrom
wirybeaver:feature/v3-dv-write-pipeline
Draft

feat: add table-aware position delete updates#919
wirybeaver wants to merge 4 commits into
apache:mainfrom
wirybeaver:feature/v3-dv-write-pipeline

Conversation

@wirybeaver

@wirybeaver wirybeaver commented Sep 5, 2026

Copy link
Copy Markdown

Summary

  • add a table-aware PositionDeleteUpdate API that resolves live data-file spec and partition metadata from the current snapshot
  • write and commit Puffin deletion vectors for v3 tables while preserving Parquet position-delete files for v2 tables
  • merge existing DVs and file-scoped Parquet position deletes, then supersede rewritten delete entries through RowDelta
  • sort v2 position deletes deterministically by file path and position, and clean failed outputs in deterministic path order
  • report cleanup deletion failures, retain ownership of undeleted outputs, and retry orphan cleanup before writing new output
  • validate referenced data files, reuse output across ordinary commit retries, and make the update terminal without deleting output after CommitStateUnknown

Tracks #916.

Dependencies

Depends on #913, #914, and #915. This draft is stacked on their prerequisite commits and should be reviewed and merged after those PRs.

Test plan

  • ./build/src/iceberg/test/data_test — 180 tests passed
  • ./build/src/iceberg/test/table_update_test --gtest_filter='RowDeltaTest.*' — 15 tests passed
  • build/precommit-venv/bin/pre-commit run --files src/iceberg/data/position_delete_update.cc src/iceberg/test/position_delete_update_test.cc — all hooks passed

Focused coverage includes descending v2 input with physical Parquet row-order verification, v2 file-scoped Parquet delete migration to a merged v3 DV, deterministic failure cleanup, ordinary commit retry reuse, and terminal output ownership after CommitStateUnknown. Cleanup-failure coverage verifies that the deletion failure is appended to the original commit error, ownership is retained for the undeleted path, and a later attempt deletes the retained orphan before writing and committing new output.

Remaining test limitation

The pipeline now opts into ValidateDataFilesExist(...).ValidateDeletedFiles(), and the focused RowDeltaTest validation suite passes. A deterministic PositionDeleteUpdate test that injects a persisted v3 rewrite/removal between DV write and retry validation is not included: the retry/replay boundary is internal to Transaction/RowDelta, and the current catalog mock seam proceeds to another catalog attempt rather than exposing a stable validation hook. Covering that exact interleaving would require broader transaction test infrastructure outside this ticket.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant