Skip to content

Backport Spark 3.5, 4.0, 4.1: Best-effort ExpireSnapshots when metadata reachable only from expired snapshots is missing - #19071

Open
mengna-lin wants to merge 2 commits into
apache:mainfrom
mengna-lin:best-effort-expire-snapshots-backport
Open

mengna-lin wants to merge 2 commits into
apache:mainfrom
mengna-lin:best-effort-expire-snapshots-backport

Conversation

@mengna-lin

@mengna-lin mengna-lin commented Nov 20, 2026 •

Copy link
Copy Markdown
Contributor

Backport of #18477 to Spark 3.5, 4.0 and 4.1. ExpireSnapshotsSparkAction fails when a manifest list or manifest reachable from an expired snapshot is already gone. The snapshots are already expired at that point, so retrying cannot make progress and orphans accumulate. Such missing metadata is now skipped and logged, and the action returns the result of the files it deleted. Missing metadata reachable from a retained snapshot still fails.

The code is identical to the Spark 4.2 change. In Spark 3.5 and 4.0, adaptive execution reports several failed stages as one exception without the failure details in its message, so the fail-fast test checks the full stack trace instead of the message.

Issue: #17715
Depends on #18477.

Test plan: TestExpireSnapshotsAction, TestDeleteReachableFilesAction and TestRemoveOrphanFilesAction pass on each version.


AI Disclosure

  • Model: Claude Sonnet 5.5
  • Platform/Tool: Claude Code
  • Human Oversight: Reviewed
  • Prompt Summary: Backport the best-effort ExpireSnapshots change from Spark 4.2 to Spark 3.5, 4.0 and 4.1.

…ble only from expired snapshots is missing

ExpireSnapshotsSparkAction fails if a manifest list or manifest
reachable from an expired snapshot is already gone. The snapshots are
already expired at that point, so retrying cannot make progress and
orphans accumulate. Skip such missing metadata when it is reachable
only from expired snapshots, log it, and return the result of the
files that were deleted.

Backport of apache#18477 from Spark 4.2.

Generated-by: Claude Code
@mengna-lin mengna-lin changed the title Spark 3.5, 4.0, 4.1: Best-effort ExpireSnapshots when metadata reachable only from expired snapshots is missing Backport Spark 3.5, 4.0, 4.1: Best-effort ExpireSnapshots when metadata reachable only from expired snapshots is missing Nov 20, 2026
@github-actions github-actions Bot added the spark label Nov 20, 2026
Adaptive execution may report several failed stages as one exception
whose message does not include the missing manifest. Check the full
stack trace, as the Spark 4.0 test does.

Generated-by: Claude Code

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant