Support rewrite_manifests table maintenance

[moomindani]

Closes #3629

Rationale for this change

Adds table.maintenance.rewrite_manifests(), the explicit maintenance counterpart of Java's Table.rewriteManifests() (Spark: rewrite_manifests). When a table accumulates many small manifests from repeated appends, scan planning slows down; this merges the current snapshot's data manifests into fewer manifests sized by commit.manifest.target-size-bytes (default 8 MB).

• Entries are rewritten as EXISTING and keep their sequence numbers and file sequence numbers.
• Delete manifests, and data manifests that need no merging, are kept as-is.
• rewrite_if(predicate) selects which data manifests to rewrite; manifests that do not match are kept as-is. Passing a predicate also disables the single-manifest shortcut, so a matching manifest is rewritten even when there is nothing to merge it with — which is what lets a caller target a specific set of manifests, such as ones written by an older library version. Contributed by @hedger9487.
• The result is committed as a replace snapshot; snapshot totals carry over unchanged (update_snapshot_summaries now handles Operation.REPLACE), and the summary reports manifests-created / manifests-kept / manifests-replaced / entries-processed, matching Java.
• V3 tables are rejected with a clear error for now: rewriting must preserve the first-row-id of rewritten manifests, which needs the read side of row lineage (Support row lineage assignment on commit #3621). The writer-side plumbing already exists in Support writing V3 manifests and manifest lists #3624.

Related: #270 tracks automatic manifest merging on write; this PR covers only the explicit maintenance API.

Related: #3124 is adding table.maintenance.compact() on the same MaintenanceTable. The two touch different methods and do not conflict; whichever lands first sets the convention for the other.

Are these changes tested?

Yes, eleven tests using an in-memory catalog with real appends: merging three manifests into one (entry counts, EXISTING status, data unchanged, replace summary with carried-over totals), sequence-number preservation including the merged manifest's min_sequence_number, single-manifest no-op, target-size grouping, and rewrites_needed(); then for rewrite_if: selective rewrite with the untouched manifests asserted to keep their exact paths and underlying data files, a single manifest rewritten because it matches, rewrites_needed() under matching and non-matching predicates, a predicate matching nothing committing nothing, a manifest holding only deleted entries left alone rather than producing an empty manifest, and plain rewrite_manifests() merging a live and a fully-deleted manifest into one. Removing the empty-manifest guard fails the fully-deleted test; removing the implementation fails all of them.

Are there any user-facing changes?

Yes: a new table.maintenance.rewrite_manifests() API with an optional rewrite_if(predicate), documented in a new "Manifest Rewriting" subsection of mkdocs/docs/api.md.

This pull request and its description were written by Claude Fable 5.

[moomindani]

Marking this ready for review.

I had kept it as a draft to land consistently with #3124, which adds compact() to the same MaintenanceTable. Revisiting that: the two PRs add separate methods and do not conflict in code, so the only thing to align is naming and argument style — which whichever lands first can set for the other. #3124 has been open since March and is itself waiting on further work, so holding this indefinitely doesn't seem useful.

State: CI green (17/17), mergeable, five tests using an in-memory catalog with real appends. V3 tables are rejected with a clear error for now, since rewriting must preserve first-row-id, which needs the read side of row lineage (#3621).

@Fokko @sungwy would either of you have time to take a look? Closes #3629.

[hedger9487]

Hi @moomindani,

Thanks for taking the initiative on this feature and setting up a great foundation!

While working on the follow-up for #3840 (suggested by @kevinjqliu to allow rewriting legacy manifests), I came across your PR. Based on the Java Iceberg reference (BaseRewriteManifests) and #3840's requirements, I noticed a few additional capabilities that might be useful:

1. Single manifest rewriting: Currently, if len(group) == 1: continue skips rewriting single manifests. For use cases like equality_ids written as list<long>, but spec and all other implementations use list<int> #3840 (upgrading legacy manifest schemas to standard types), users need to be able to rewrite/upgrade single manifests as well.
2. Selective filtering (rewrite_if): Adding rewrite_if(predicate) from Java's RewriteManifests.rewriteIf() allows users to selectively target specific manifests (e.g. only legacy manifests or specific partitions).
3. Bin-packing & Concurrency: Reusing PyIceberg's ListPacker and ExecutorFactory allows parallel manifest writes for large tables.
4. Safety guard: Adding _validate_files_counts ensures zero data files are dropped during the rewrite.

I already have a tested implementation with these enhancements ready locally. If you're open to it, I'd be more than happy to collaborate by opening a PR directly to your branch or co-authoring to help land this together!

Thanks again for your great work!

[moomindani]

@hedger9487 thanks for the careful read. One piece of context that shapes my answer: in @kevinjqliu's plan on #3840 (31 August), step 3 is exactly this API — a maintenance operation that rewrites the manifests of the current snapshot so the legacy equality_ids: list<long> schema gets re-emitted as int. So I want to keep this PR small enough to land, and pull in the piece that plan needs.

Of your four, the one I want here is (1) — and I think your (2) is the right way to express it, rather than a second feature. Today rewrite_manifests() takes no arguments at all, and _group_by_target_size short-circuits single-manifest groups into kept_manifests, so "rewrite a manifest that does not need size merging" cannot be requested through the public API. Java says that with RewriteManifests.rewriteIf(Predicate<ManifestFile>), and #3840 wants to target precisely the legacy manifests, so a predicate fits the use case better than an unconditional force would. If your local implementation already has it in that shape, that is the change I would take.

(3) bin-packing and concurrency via ListPacker / ExecutorFactory, and (4) _validate_files_counts, I would rather do as follow-ups. Both are worth having, and neither is needed by #3840; this PR has been open since 8 July without a review, and I do not want to grow it while it waits. I will review those promptly once this lands, and I am happy for them to be queued against it in the meantime.

A PR against my branch is very welcome — moomindani:moomindani/rewrite-manifests — and you will be co-author either way. If you would rather I write the predicate change myself from your description, that works too; tell me which you prefer.

@kevinjqliu one thing worth flagging from your #3840 plan: step 3 is this PR. CI is green and it has been open since 8 July with no review, and with the rewrite_if predicate above it covers the "users can correct the avro files" step directly. Would you have time to look, or is there someone you would rather route it to?

[hedger9487]

Hi @moomindani,

Thank you for the helpful context regarding #3840!

Following our discussion, I have opened a PR against your branch with the rewrite_if implementation and unit tests:
👉 moomindani#1

Summary of changes:

1. Added rewrite_if(predicate: Callable[[ManifestFile], bool]) -> RewriteManifests to RewriteManifests.
2. When a predicate is provided, matching manifests are rewritten even for single-manifest groups (len(group) == 1), allowing schema upgrades for equality_ids written as list<long>, but spec and all other implementations use list<int> #3840. Manifests not matching the predicate are kept as-is.
3. Updated rewrites_needed() to evaluate the predicate when provided.
4. Added unit tests covering selective filtering, single-manifest predicate rewrites, and rewrites_needed.

Please take a look when you have a moment and let me know if there is anything you would like adjusted!