Repository navigation
Add ExpireSnapshots.retain_last - #4437
rambleraptor wants to merge 1 commit into
Conversation
Port Java's ExpireSnapshots.retainLast: snapshots that older_than would expire are kept if they are among the N most recent ancestors of a branch. Snapshots expired by ID are unaffected.
142a52d to
0b81a7f
Compare
| self._snapshot_ids_older_than.add(snapshot.snapshot_id) | ||
| return self | ||
|
|
||
| def retain_last(self, num_snapshots: int) -> ExpireSnapshots: |
There was a problem hiding this comment.
In Java, retainLast(n) alone still expires snapshots older than history.expire.max-snapshot-age-ms, keeping n per branch (RemoveSnapshots, test: apache/iceberg#19049). Here it does nothing without older_than.
Could we note that in the docstring? Honoring history.expire.max-snapshot-age-ms and history.expire.min-snapshots-to-keep (and the ref-level overrides) looks like a separate follow-up.
| assert {snapshot.snapshot_id for snapshot in txn.table_metadata.snapshots} == {2, 3, 4, 5} | ||
|
|
||
|
|
||
| def test_expire_snapshots_retain_last_does_not_retain_by_id(example_table_metadata_v2: dict[str, Any]) -> None: |
There was a problem hiding this comment.
Could we add a test that retain_last alone expires nothing? It pins the current behavior, so it's a visible change if the history.expire.* defaults get wired in later.
def test_expire_snapshots_retain_last_without_older_than(example_table_metadata_v2: dict[str, Any]) -> None:
table = _table_with_branches(example_table_metadata_v2)
txn = table.transaction()
ExpireSnapshots(txn).retain_last(1).commit()
assert {snapshot.snapshot_id for snapshot in txn.table_metadata.snapshots} == {1, 2, 3, 4, 5}
Fokko
left a comment
There was a problem hiding this comment.
Thanks @rambleraptor! Nice addition, the chaining example in the docs reads well.
Verified locally: all 17 tests in tests/table/test_expire_snapshots.py pass, CI is green.
One small thing I checked and is fine: by_id snapshots still expire even with retain_last, since they are added directly to _snapshot_ids_to_expire. Also like that the ordering of older_than/retain_last does not matter since the union happens in _commit.
LGTM!
Rationale for this change
This helper method lets you retain the N most recent ancestors of a branch. I've included some doc examples of how you can use this for chaining multiple commands together.
Are these changes tested?
Unit tests included
Are there any user-facing changes?
retain_lastmethod to ExpireSnapshots.