Repository navigation
Core: Carry forward custom snapshot properties through replace commits - #18401
anuragmantri wants to merge 2 commits into
Conversation
mxm
left a comment
There was a problem hiding this comment.
Thanks for the PR @anuragmantri! I wonder, why can readers not traverse the snapshot history, e.g. like Flink's IcebergSink does, to discover their keys?
| for (Map.Entry<String, String> entry : previousSummary.entrySet()) { | ||
| String key = entry.getKey(); | ||
| if (!RESERVED_PROPERTIES.contains(key) | ||
| && !key.startsWith(CHANGED_PARTITION_PREFIX) | ||
| && !environment.containsKey(key) | ||
| && !summary.containsKey(key)) { | ||
| carried.put(key, entry.getValue()); | ||
| } | ||
| } |
There was a problem hiding this comment.
I'm not 100% sure about the implications this might have. The meaning of snapshot summary changes when we copy the last summary into the REPLACE commit.
There was a problem hiding this comment.
That's a fair concern about the semantics here. Even after my new change to allow list, this concern still holds. My thought here is that the custom properties are usually the ones that hold state of a table (for example (ingested-by: service). With the allow list, the table owner opt into these properties to be carried forward by replace. So what the parent's property said about the table is still true since replace operations don't change the data in a table.
There was a problem hiding this comment.
The example you gave has the same issue. When there are multiple writers, we would take the last ingested-by value, which may be misleading when we replace multiple snapshots written by different services. However, I understand that it might be useful to save some properties.
There was a problem hiding this comment.
I think the PR copies the properties of the last commit, and it has nothing to do with the commits which the REPLACE commit actually updates. Am I right?
There was a problem hiding this comment.
Yes, that's right. It copies the listed properties from the head of the branch the replace commits to, not from the commits that wrote the rewritten files. That's deliberate. The listed keys are meant to describe the table as of its current head, and a replace commit doesn't change the table's contents, so the head's values still apply.
There was a problem hiding this comment.
Could you please update the description based on this?
After that, I would like to ask you to send a letter to the dev list.
Why I see this feature which is sometimes useful, I feel that a similar goal could be archived with good configuration and without introducing a new config.
There could be some other use-case which could not be served without this config. If so, it would be good to see them.
Thanks,
Peter
| private static final Set<String> RESERVED_PROPERTIES = | ||
| ImmutableSet.of( | ||
| ADDED_FILES_PROP, | ||
| DELETED_FILES_PROP, |
There was a problem hiding this comment.
I wonder if this should be an allow list rather than a static deny list?
There was a problem hiding this comment.
Agreed, I switched to an allow list. The new table property write.summary.carry-forward-keys lists the custom properties to copy, and nothing is copied unless it's set. That also removed the reserved-key list entirely. Anything the commit itself sets, including Iceberg's computed metrics, is never overwritten.
| public static final boolean MANIFEST_MERGE_ENABLED_DEFAULT = true; | ||
|
|
||
| public static final String SNAPSHOT_PROPERTY_CARRY_FORWARD_ENABLED = | ||
| "commit.snapshot-property-carry-forward.enabled"; |
There was a problem hiding this comment.
suggestion: I believe the terminology is snapshot-summary instead of snapshot-property. so maybe consider write.summary.carry-forward, similar to existing WRITE_PARTITION_SUMMARY_LIMIT
There was a problem hiding this comment.
Thanks, renamed it to write.summary.carry-forward-keys, next to WRITE_PARTITION_SUMMARY_LIMIT. It now takes a list of keys, since it became an allow list.
| Map<String, String> carried = Maps.newHashMap(); | ||
| for (Map.Entry<String, String> entry : previousSummary.entrySet()) { | ||
| String key = entry.getKey(); | ||
| if (!RESERVED_PROPERTIES.contains(key) | ||
| && !key.startsWith(CHANGED_PARTITION_PREFIX) | ||
| && !environment.containsKey(key) | ||
| && !summary.containsKey(key)) { | ||
| carried.put(key, entry.getValue()); | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
I think we can use built-in, also easier to see the predicate this way on which key to carry forward with
| Map<String, String> carried = Maps.newHashMap(); | |
| for (Map.Entry<String, String> entry : previousSummary.entrySet()) { | |
| String key = entry.getKey(); | |
| if (!RESERVED_PROPERTIES.contains(key) | |
| && !key.startsWith(CHANGED_PARTITION_PREFIX) | |
| && !environment.containsKey(key) | |
| && !summary.containsKey(key)) { | |
| carried.put(key, entry.getValue()); | |
| } | |
| } | |
| return PropertyUtil.filterProperties( | |
| previousSummary, | |
| key -> | |
| !RESERVED_PROPERTIES.contains(key) | |
| && !key.startsWith(CHANGED_PARTITION_PREFIX) | |
| && !environment.containsKey(key) | |
| && !summary.containsKey(key)); | |
There was a problem hiding this comment.
Thanks for the suggestion. This code is gone now that the property is an allow list, so there's no reserved-key predicate left to filter on.
| Map<String, String> environment = EnvironmentContext.get(); | ||
| ImmutableMap.Builder<String, String> builder = ImmutableMap.builder(); | ||
|
|
||
| if (DataOperations.REPLACE.equals(operation()) |
There was a problem hiding this comment.
I think this expand the current scope, does it warrant a update to existing comment on LINE 391
/** Returns the snapshot summary from the implementation and updates totals. */?
There was a problem hiding this comment.
Good catch, I updated that Javadoc to mention the properties carried forward for replace commits.
| } | ||
|
|
||
| @TestTemplate | ||
| void appendDoesNotCarryForwardProperties() { |
There was a problem hiding this comment.
seems a bit inconsistent if we do conditional carry forward based on operation type
There was a problem hiding this comment.
This PR Is scoped only to replace commits. A replace commit doesn't change the table's data, so whatever the parent's properties had just before the replace commit will be carried forward.
Writers that change data are expected to set the property themselves.
42d8c76 to
94c6c76
Compare
There was a problem hiding this comment.
Thanks for the review @mxm and @dramaticlly! I've switched to an allow list, details inline.
On traversing the snapshot history: that works as long as the snapshot that set the key is still around, which is what Flink's sink relies on to find its last committed checkpoint. On tables that see only compactions for a while, snapshot expiration can remove the last snapshot that still had the key, and after that nothing in the history has it. Copying the value onto each replace snapshot keeps it on the current snapshot regardless of expiration. A non-replace commit that doesn't set the key still ends the chain. I think that's the right boundary, since those commits can change the data the property describes.
This is ready for another review.
| private static final Set<String> RESERVED_PROPERTIES = | ||
| ImmutableSet.of( | ||
| ADDED_FILES_PROP, | ||
| DELETED_FILES_PROP, |
There was a problem hiding this comment.
Agreed, I switched to an allow list. The new table property write.summary.carry-forward-keys lists the custom properties to copy, and nothing is copied unless it's set. That also removed the reserved-key list entirely. Anything the commit itself sets, including Iceberg's computed metrics, is never overwritten.
| for (Map.Entry<String, String> entry : previousSummary.entrySet()) { | ||
| String key = entry.getKey(); | ||
| if (!RESERVED_PROPERTIES.contains(key) | ||
| && !key.startsWith(CHANGED_PARTITION_PREFIX) | ||
| && !environment.containsKey(key) | ||
| && !summary.containsKey(key)) { | ||
| carried.put(key, entry.getValue()); | ||
| } | ||
| } |
There was a problem hiding this comment.
That's a fair concern about the semantics here. Even after my new change to allow list, this concern still holds. My thought here is that the custom properties are usually the ones that hold state of a table (for example (ingested-by: service). With the allow list, the table owner opt into these properties to be carried forward by replace. So what the parent's property said about the table is still true since replace operations don't change the data in a table.
| public static final boolean MANIFEST_MERGE_ENABLED_DEFAULT = true; | ||
|
|
||
| public static final String SNAPSHOT_PROPERTY_CARRY_FORWARD_ENABLED = | ||
| "commit.snapshot-property-carry-forward.enabled"; |
There was a problem hiding this comment.
Thanks, renamed it to write.summary.carry-forward-keys, next to WRITE_PARTITION_SUMMARY_LIMIT. It now takes a list of keys, since it became an allow list.
| Map<String, String> carried = Maps.newHashMap(); | ||
| for (Map.Entry<String, String> entry : previousSummary.entrySet()) { | ||
| String key = entry.getKey(); | ||
| if (!RESERVED_PROPERTIES.contains(key) | ||
| && !key.startsWith(CHANGED_PARTITION_PREFIX) | ||
| && !environment.containsKey(key) | ||
| && !summary.containsKey(key)) { | ||
| carried.put(key, entry.getValue()); | ||
| } | ||
| } | ||
|
|
There was a problem hiding this comment.
Thanks for the suggestion. This code is gone now that the property is an allow list, so there's no reserved-key predicate left to filter on.
| Map<String, String> environment = EnvironmentContext.get(); | ||
| ImmutableMap.Builder<String, String> builder = ImmutableMap.builder(); | ||
|
|
||
| if (DataOperations.REPLACE.equals(operation()) |
There was a problem hiding this comment.
Good catch, I updated that Javadoc to mention the properties carried forward for replace commits.
| } | ||
|
|
||
| @TestTemplate | ||
| void appendDoesNotCarryForwardProperties() { |
There was a problem hiding this comment.
This PR Is scoped only to replace commits. A replace commit doesn't change the table's data, so whatever the parent's properties had just before the replace commit will be carried forward.
Writers that change data are expected to set the property themselves.
mxm
left a comment
There was a problem hiding this comment.
On traversing the snapshot history: that works as long as the snapshot that set the key is still around, which is what Flink's sink relies on to find its last committed checkpoint. On tables that see only compactions for a while, snapshot expiration can remove the last snapshot that still had the key, and after that nothing in the history has it. Copying the value onto each replace snapshot keeps it on the current snapshot regardless of expiration. A non-replace commit that doesn't set the key still ends the chain. I think that's the right boundary, since those commits can change the data the property describes.
Snapshot expiry could be configured in a way to preserve at least one APPEND commit. On the other hand, that might render the compaction useless in some scenarios because you need to hold on to that one commit which holds the uncompacted data files you are trying to get rid of.
Overall, I do think that this change makes sense because it helps to preserve certain snapshot summary properties which would otherwise get lost with compaction enabled. Since it's a core change affecting all writers, it might still be worthwhile to discuss on the dev mailing list.
| return withCarriedForwardProperties(previous, previousSummary, builder.build()); | ||
| } | ||
|
|
||
| private Map<String, String> withCarriedForwardProperties( |
There was a problem hiding this comment.
This name is a bit confusing because we already carry-forward other built-in props. How about:
| private Map<String, String> withCarriedForwardProperties( | |
| private Map<String, String> carryForwardCustomProperties( |
|
|
||
| builder.putAll(EnvironmentContext.get()); | ||
| return builder.build(); | ||
| return withCarriedForwardProperties(previous, previousSummary, builder.build()); |
There was a problem hiding this comment.
I thought more about this and discussed with @pvary. We wonder, does this affect the spec? SnapshotProducer will affect all Java writers, but what about other implementations? This feature is only useful if all writers respect it.
It may be useful to discuss this on the dev mailing list first.
94c6c76 to
ef37b49
Compare
|
Thanks @mxm and @pvary. I researched a bit more on the previous issues and PRs related to similar requests. I don't think this touches the spec. When the optional summary fields were standardized in #11660, they went into Appendix F as implementation guidance. @RussellSpitzer also asked to leave out anything tied directly to a table property (and see this comment in #16734 on keeping table properties out of the spec). Summary already carries keys that describe the writer or the table's state rather than the changes in that commit. For example, Appendix F lists Other implementations that don't support it will just drop the listed keys on replace, which is today's behavior. Also, I'm still open to posting in @dev if you still think that is necessary. |
|
Thanks @pvary. I updated the description, and the config docs and Javadoc now say the values come from the current head of the branch, not from the commits that wrote the rewritten files. I'll send the email to dev@ shortly. On your question about config alone, the cases I can think of are:
Did you have anything else in mind? |
I think depending on the head's summary is brittle. Any concurrent write or compaction could break the use-case.
This use-case could be server by a compaction only feature. We don't need to be this in the core. |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
Generated-by: Claude Code (Claude Opus 5.5)
ef37b49 to
f3d54e4
Compare
Maintenance commits such as rewrite data files and rewrite manifests build their snapshot summary only from the operation, so custom properties written by the producer of the replaced snapshot are lost after compaction. This adds an opt-in table property,
write.summary.carry-forward-keys, a comma-separated list of custom summary properties that replace commits copy from the branch head they replace. Unlisted properties are never copied. Properties set on the commit itself, including Iceberg's computed metrics, take precedence. The values describe the table at the branch head, not the files being rewritten.Test plan: Core tests in
TestSnapshotSummary, across format versions 1–3AI Disclosure