Skip to content

Core: Carry forward custom snapshot properties through replace commits - #18401

Open
anuragmantri wants to merge 2 commits into
apache:mainfrom
anuragmantri:carry-forward-snapshot-properties
Open

anuragmantri wants to merge 2 commits into
apache:mainfrom
anuragmantri:carry-forward-snapshot-properties

Conversation

@anuragmantri

@anuragmantri anuragmantri commented Oct 6, 2026 •

Copy link
Copy Markdown
Collaborator

Maintenance commits such as rewrite data files and rewrite manifests build their snapshot summary only from the operation, so custom properties written by the producer of the replaced snapshot are lost after compaction. This adds an opt-in table property, write.summary.carry-forward-keys, a comma-separated list of custom summary properties that replace commits copy from the branch head they replace. Unlisted properties are never copied. Properties set on the commit itself, including Iceberg's computed metrics, take precedence. The values describe the table at the branch head, not the files being rewritten.

Test plan: Core tests in TestSnapshotSummary, across format versions 1–3

  • Replace commits copy listed properties only, and nothing when the table property is unset
  • Explicitly set properties win
  • Properties computed by the commit are not overwritten by copied values
  • Values come from the target branch head at commit time
  • Appends don't inherit

AI Disclosure

  • Model: Claude Opus 5.5
  • Platform/Tool: Claude Code
  • Human Oversight: fully reviewed
  • Prompt Summary: Make Iceberg replace commits (compaction, manifest rewrites) optionally keep the replaced snapshot's custom summary properties, using a table-level allow list.

@anuragmantri

Copy link
Copy Markdown
Collaborator Author

@mxm @nastra - Could you please provide a review? Thanks!

@mxm mxm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the PR @anuragmantri! I wonder, why can readers not traverse the snapshot history, e.g. like Flink's IcebergSink does, to discover their keys?

Comment on lines +127 to +135
for (Map.Entry<String, String> entry : previousSummary.entrySet()) {
String key = entry.getKey();
if (!RESERVED_PROPERTIES.contains(key)
&& !key.startsWith(CHANGED_PARTITION_PREFIX)
&& !environment.containsKey(key)
&& !summary.containsKey(key)) {
carried.put(key, entry.getValue());
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not 100% sure about the implications this might have. The meaning of snapshot summary changes when we copy the last summary into the REPLACE commit.

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a fair concern about the semantics here. Even after my new change to allow list, this concern still holds. My thought here is that the custom properties are usually the ones that hold state of a table (for example (ingested-by: service). With the allow list, the table owner opt into these properties to be carried forward by replace. So what the parent's property said about the table is still true since replace operations don't change the data in a table.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The example you gave has the same issue. When there are multiple writers, we would take the last ingested-by value, which may be misleading when we replace multiple snapshots written by different services. However, I understand that it might be useful to save some properties.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think the PR copies the properties of the last commit, and it has nothing to do with the commits which the REPLACE commit actually updates. Am I right?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, that's right. It copies the listed properties from the head of the branch the replace commits to, not from the commits that wrote the rewritten files. That's deliberate. The listed keys are meant to describe the table as of its current head, and a replace commit doesn't change the table's contents, so the head's values still apply.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you please update the description based on this?

After that, I would like to ask you to send a letter to the dev list.
Why I see this feature which is sometimes useful, I feel that a similar goal could be archived with good configuration and without introducing a new config.

There could be some other use-case which could not be served without this config. If so, it would be good to see them.

Thanks,
Peter

private static final Set<String> RESERVED_PROPERTIES =
ImmutableSet.of(
ADDED_FILES_PROP,
DELETED_FILES_PROP,

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder if this should be an allow list rather than a static deny list?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, I switched to an allow list. The new table property write.summary.carry-forward-keys lists the custom properties to copy, and nothing is copied unless it's set. That also removed the reserved-key list entirely. Anything the commit itself sets, including Iceberg's computed metrics, is never overwritten.

public static final boolean MANIFEST_MERGE_ENABLED_DEFAULT = true;

public static final String SNAPSHOT_PROPERTY_CARRY_FORWARD_ENABLED =
"commit.snapshot-property-carry-forward.enabled";

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: I believe the terminology is snapshot-summary instead of snapshot-property. so maybe consider write.summary.carry-forward, similar to existing WRITE_PARTITION_SUMMARY_LIMIT

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, renamed it to write.summary.carry-forward-keys, next to WRITE_PARTITION_SUMMARY_LIMIT. It now takes a list of keys, since it became an allow list.

Comment on lines +126 to +136
Map<String, String> carried = Maps.newHashMap();
for (Map.Entry<String, String> entry : previousSummary.entrySet()) {
String key = entry.getKey();
if (!RESERVED_PROPERTIES.contains(key)
&& !key.startsWith(CHANGED_PARTITION_PREFIX)
&& !environment.containsKey(key)
&& !summary.containsKey(key)) {
carried.put(key, entry.getValue());
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we can use built-in, also easier to see the predicate this way on which key to carry forward with

Suggested change
Map<String, String> carried = Maps.newHashMap();
for (Map.Entry<String, String> entry : previousSummary.entrySet()) {
String key = entry.getKey();
if (!RESERVED_PROPERTIES.contains(key)
&& !key.startsWith(CHANGED_PARTITION_PREFIX)
&& !environment.containsKey(key)
&& !summary.containsKey(key)) {
carried.put(key, entry.getValue());
}
}
return PropertyUtil.filterProperties(
previousSummary,
key ->
!RESERVED_PROPERTIES.contains(key)
&& !key.startsWith(CHANGED_PARTITION_PREFIX)
&& !environment.containsKey(key)
&& !summary.containsKey(key));

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion. This code is gone now that the property is an allow list, so there's no reserved-key predicate left to filter on.

Map<String, String> environment = EnvironmentContext.get();
ImmutableMap.Builder<String, String> builder = ImmutableMap.builder();

if (DataOperations.REPLACE.equals(operation())

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this expand the current scope, does it warrant a update to existing comment on LINE 391
/** Returns the snapshot summary from the implementation and updates totals. */?

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, I updated that Javadoc to mention the properties carried forward for replace commits.

}

@TestTemplate
void appendDoesNotCarryForwardProperties() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

seems a bit inconsistent if we do conditional carry forward based on operation type

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR Is scoped only to replace commits. A replace commit doesn't change the table's data, so whatever the parent's properties had just before the replace commit will be carried forward.

Writers that change data are expected to set the property themselves.

@anuragmantri
anuragmantri force-pushed the carry-forward-snapshot-properties branch from 42d8c76 to 94c6c76 Compare October 7, 2026 23:01

@anuragmantri anuragmantri left a comment •

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review @mxm and @dramaticlly! I've switched to an allow list, details inline.

On traversing the snapshot history: that works as long as the snapshot that set the key is still around, which is what Flink's sink relies on to find its last committed checkpoint. On tables that see only compactions for a while, snapshot expiration can remove the last snapshot that still had the key, and after that nothing in the history has it. Copying the value onto each replace snapshot keeps it on the current snapshot regardless of expiration. A non-replace commit that doesn't set the key still ends the chain. I think that's the right boundary, since those commits can change the data the property describes.

This is ready for another review.

private static final Set<String> RESERVED_PROPERTIES =
ImmutableSet.of(
ADDED_FILES_PROP,
DELETED_FILES_PROP,

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, I switched to an allow list. The new table property write.summary.carry-forward-keys lists the custom properties to copy, and nothing is copied unless it's set. That also removed the reserved-key list entirely. Anything the commit itself sets, including Iceberg's computed metrics, is never overwritten.

Comment on lines +127 to +135
for (Map.Entry<String, String> entry : previousSummary.entrySet()) {
String key = entry.getKey();
if (!RESERVED_PROPERTIES.contains(key)
&& !key.startsWith(CHANGED_PARTITION_PREFIX)
&& !environment.containsKey(key)
&& !summary.containsKey(key)) {
carried.put(key, entry.getValue());
}
}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a fair concern about the semantics here. Even after my new change to allow list, this concern still holds. My thought here is that the custom properties are usually the ones that hold state of a table (for example (ingested-by: service). With the allow list, the table owner opt into these properties to be carried forward by replace. So what the parent's property said about the table is still true since replace operations don't change the data in a table.

public static final boolean MANIFEST_MERGE_ENABLED_DEFAULT = true;

public static final String SNAPSHOT_PROPERTY_CARRY_FORWARD_ENABLED =
"commit.snapshot-property-carry-forward.enabled";

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks, renamed it to write.summary.carry-forward-keys, next to WRITE_PARTITION_SUMMARY_LIMIT. It now takes a list of keys, since it became an allow list.

Comment on lines +126 to +136
Map<String, String> carried = Maps.newHashMap();
for (Map.Entry<String, String> entry : previousSummary.entrySet()) {
String key = entry.getKey();
if (!RESERVED_PROPERTIES.contains(key)
&& !key.startsWith(CHANGED_PARTITION_PREFIX)
&& !environment.containsKey(key)
&& !summary.containsKey(key)) {
carried.put(key, entry.getValue());
}
}

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion. This code is gone now that the property is an allow list, so there's no reserved-key predicate left to filter on.

Map<String, String> environment = EnvironmentContext.get();
ImmutableMap.Builder<String, String> builder = ImmutableMap.builder();

if (DataOperations.REPLACE.equals(operation())

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Good catch, I updated that Javadoc to mention the properties carried forward for replace commits.

}

@TestTemplate
void appendDoesNotCarryForwardProperties() {

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This PR Is scoped only to replace commits. A replace commit doesn't change the table's data, so whatever the parent's properties had just before the replace commit will be carried forward.

Writers that change data are expected to set the property themselves.

@mxm mxm left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On traversing the snapshot history: that works as long as the snapshot that set the key is still around, which is what Flink's sink relies on to find its last committed checkpoint. On tables that see only compactions for a while, snapshot expiration can remove the last snapshot that still had the key, and after that nothing in the history has it. Copying the value onto each replace snapshot keeps it on the current snapshot regardless of expiration. A non-replace commit that doesn't set the key still ends the chain. I think that's the right boundary, since those commits can change the data the property describes.

Snapshot expiry could be configured in a way to preserve at least one APPEND commit. On the other hand, that might render the compaction useless in some scenarios because you need to hold on to that one commit which holds the uncompacted data files you are trying to get rid of.

Overall, I do think that this change makes sense because it helps to preserve certain snapshot summary properties which would otherwise get lost with compaction enabled. Since it's a core change affecting all writers, it might still be worthwhile to discuss on the dev mailing list.

return withCarriedForwardProperties(previous, previousSummary, builder.build());
}

private Map<String, String> withCarriedForwardProperties(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This name is a bit confusing because we already carry-forward other built-in props. How about:

Suggested change
private Map<String, String> withCarriedForwardProperties(
private Map<String, String> carryForwardCustomProperties(


builder.putAll(EnvironmentContext.get());
return builder.build();
return withCarriedForwardProperties(previous, previousSummary, builder.build());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I thought more about this and discussed with @pvary. We wonder, does this affect the spec? SnapshotProducer will affect all Java writers, but what about other implementations? This feature is only useful if all writers respect it.

It may be useful to discuss this on the dev mailing list first.

@anuragmantri
anuragmantri force-pushed the carry-forward-snapshot-properties branch from 94c6c76 to ef37b49 Compare October 8, 2026 20:43
@anuragmantri

Copy link
Copy Markdown
Collaborator Author

Thanks @mxm and @pvary. I researched a bit more on the previous issues and PRs related to similar requests. I don't think this touches the spec.

When the optional summary fields were standardized in #11660, they went into Appendix F as implementation guidance. @RussellSpitzer also asked to leave out anything tied directly to a table property (and see this comment in #16734 on keeping table properties out of the spec).

Summary already carries keys that describe the writer or the table's state rather than the changes in that commit. For example, Appendix F lists engine-name and engine-version under "Other Fields" alongside the WAP and cherry-pick markers. The keys this PR copies fall under that category of properties the table owner explicitly lists as describing the table, copied only by replace commits that don't change the table's data.

Other implementations that don't support it will just drop the listed keys on replace, which is today's behavior. Also, PyIceberg and iceberg-rust don't produce replace commits yet IIUC.

I'm still open to posting in @dev if you still think that is necessary.

@anuragmantri

Copy link
Copy Markdown
Collaborator Author

Thanks @pvary. I updated the description, and the config docs and Javadoc now say the values come from the current head of the branch, not from the commits that wrote the rewritten files.

I'll send the email to dev@ shortly.

On your question about config alone, the cases I can think of are:

  • Letting each maintenance job read the head's summary and set the keys itself works when the job controls its own code. It doesn't work for CALL system.rewrite_data_files or rewrite_manifests from SQL, or for a generic maintenance service that doesn't know the writers' keys.
  • Retaining old snapshots with expiration settings keeps the uncompacted data files alive, which defeats the purpose of compaction.

Did you have anything else in mind?

@pvary

pvary commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

Letting each maintenance job read the head's summary and set the keys itself works when the job controls its own code. It doesn't work for CALL system.rewrite_data_files or rewrite_manifests from SQL, or for a generic maintenance service that doesn't know the writers' keys.

I think depending on the head's summary is brittle. Any concurrent write or compaction could break the use-case.

Retaining old snapshots with expiration settings keeps the uncompacted data files alive, which defeats the purpose of compaction.

This use-case could be server by a compaction only feature. We don't need to be this in the core.

@github-actions

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Nov 20, 2026
@anuragmantri
anuragmantri force-pushed the carry-forward-snapshot-properties branch from ef37b49 to f3d54e4 Compare November 20, 2026 21:26
@github-actions github-actions Bot removed the stale label Nov 21, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants