Skip to content

Add ExpireSnapshots.retain_last - #4437

Open
rambleraptor wants to merge 1 commit into
apache:mainfrom
rambleraptor:expire-snapshots-retain-last
Open

rambleraptor wants to merge 1 commit into
apache:mainfrom
rambleraptor:expire-snapshots-retain-last

Conversation

@rambleraptor

@rambleraptor rambleraptor commented Nov 19, 2026 •

Copy link
Copy Markdown
Collaborator

Rationale for this change

This helper method lets you retain the N most recent ancestors of a branch. I've included some doc examples of how you can use this for chaining multiple commands together.

Are these changes tested?

Unit tests included

Are there any user-facing changes?

  • Added retain_last method to ExpireSnapshots.

Port Java's ExpireSnapshots.retainLast: snapshots that older_than would
expire are kept if they are among the N most recent ancestors of a
branch. Snapshots expired by ID are unaffected.
@rambleraptor
rambleraptor requested a review from Fokko November 19, 2026 23:25
@rambleraptor
rambleraptor force-pushed the expire-snapshots-retain-last branch from 142a52d to 0b81a7f Compare November 19, 2026 23:25
self._snapshot_ids_older_than.add(snapshot.snapshot_id)
return self

def retain_last(self, num_snapshots: int) -> ExpireSnapshots:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In Java, retainLast(n) alone still expires snapshots older than history.expire.max-snapshot-age-ms, keeping n per branch (RemoveSnapshots, test: apache/iceberg#19049). Here it does nothing without older_than.

Could we note that in the docstring? Honoring history.expire.max-snapshot-age-ms and history.expire.min-snapshots-to-keep (and the ref-level overrides) looks like a separate follow-up.

assert {snapshot.snapshot_id for snapshot in txn.table_metadata.snapshots} == {2, 3, 4, 5}


def test_expire_snapshots_retain_last_does_not_retain_by_id(example_table_metadata_v2: dict[str, Any]) -> None:

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add a test that retain_last alone expires nothing? It pins the current behavior, so it's a visible change if the history.expire.* defaults get wired in later.

def test_expire_snapshots_retain_last_without_older_than(example_table_metadata_v2: dict[str, Any]) -> None:
    table = _table_with_branches(example_table_metadata_v2)

    txn = table.transaction()
    ExpireSnapshots(txn).retain_last(1).commit()

    assert {snapshot.snapshot_id for snapshot in txn.table_metadata.snapshots} == {1, 2, 3, 4, 5}

@Fokko Fokko left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @rambleraptor! Nice addition, the chaining example in the docs reads well.

Verified locally: all 17 tests in tests/table/test_expire_snapshots.py pass, CI is green.

One small thing I checked and is fine: by_id snapshots still expire even with retain_last, since they are added directly to _snapshot_ids_to_expire. Also like that the ordering of older_than/retain_last does not matter since the union happens in _commit.

LGTM!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants