Repository navigation
API, Core: Add Snapshot.rootManifestLocation() #17523
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes from all commits
dcdacd7
197d674
0d47d5c
f55844a
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -241,8 +241,8 @@ private static List<Snapshot> updatePathInSnapshots( | |
| TableMetadata metadata, String sourcePrefix, String targetPrefix) { | ||
| List<Snapshot> newSnapshots = Lists.newArrayListWithCapacity(metadata.snapshots().size()); | ||
| for (Snapshot snapshot : metadata.snapshots()) { | ||
| String newManifestListLocation = | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. there are a few more places in this class that reference snapshot.manifestListLocation() directly, should we update those as well?
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I don't think that is necessary. This is updating this method to fulfill its contract, not going down the rabbit hole of making relocation work in v4. |
||
| newPath(snapshot.manifestListLocation(), sourcePrefix, targetPrefix); | ||
| String manifestList = snapshot.manifestListLocation(); | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. do we also need to handle ReachableFileCleanup and IncrementalFileCleanup?
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'd keep that out of this PR since nothing writes root manifests yet. Both cleanup paths read through allManifests, which now throws UnsupportedOperationException for root-manifest snapshots, so they fail loudly instead of silently skipping files.
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. We're introducing the new field to
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I think it's reasonable to include this. This behavior is correct for this method's contract. We don't need to make the whole table rewrite work, but where we can update methods to fully implement their guarantees it seems like a good idea. |
||
| String rootManifest = snapshot.rootManifestLocation(); | ||
| Snapshot newSnapshot = | ||
| new BaseSnapshot( | ||
| snapshot.sequenceNumber(), | ||
|
|
@@ -252,7 +252,8 @@ private static List<Snapshot> updatePathInSnapshots( | |
| snapshot.operation(), | ||
| snapshot.summary(), | ||
| snapshot.schemaId(), | ||
| newManifestListLocation, | ||
| manifestList == null ? null : newPath(manifestList, sourcePrefix, targetPrefix), | ||
| rootManifest == null ? null : newPath(rootManifest, sourcePrefix, targetPrefix), | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. we should probably have a test in
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. added |
||
| snapshot.firstRowId(), | ||
| snapshot.addedRows(), | ||
| snapshot.keyId()); | ||
|
|
||
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -50,6 +50,7 @@ private SnapshotParser() {} | |
| private static final String OPERATION = "operation"; | ||
| private static final String MANIFESTS = "manifests"; | ||
| private static final String MANIFEST_LIST = "manifest-list"; | ||
| private static final String ROOT_MANIFEST = "root-manifest"; | ||
| private static final String SCHEMA_ID = "schema-id"; | ||
| private static final String FIRST_ROW_ID = "first-row-id"; | ||
| private static final String ADDED_ROWS = "added-rows"; | ||
|
|
@@ -83,7 +84,10 @@ static void toJson(Snapshot snapshot, JsonGenerator generator) throws IOExceptio | |
| } | ||
|
|
||
| String manifestList = snapshot.manifestListLocation(); | ||
| if (manifestList != null) { | ||
| String rootManifest = snapshot.rootManifestLocation(); | ||
| if (rootManifest != null) { | ||
| generator.writeStringField(ROOT_MANIFEST, rootManifest); | ||
| } else if (manifestList != null) { | ||
| // write just the location. manifests should not be embedded in JSON along with a list | ||
| generator.writeStringField(MANIFEST_LIST, manifestList); | ||
| } else { | ||
|
|
@@ -176,9 +180,10 @@ static Snapshot fromJson(JsonNode node) { | |
|
|
||
| String keyId = JsonUtil.getStringOrNull(KEY_ID, node); | ||
|
|
||
| if (node.has(MANIFEST_LIST)) { | ||
| // the manifest list is stored in a manifest list file | ||
| String manifestList = JsonUtil.getString(MANIFEST_LIST, node); | ||
| String manifestList = JsonUtil.getStringOrNull(MANIFEST_LIST, node); | ||
| String rootManifest = JsonUtil.getStringOrNull(ROOT_MANIFEST, node); | ||
|
|
||
| if (manifestList != null || rootManifest != null) { | ||
| return new BaseSnapshot( | ||
|
Contributor
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This seems another reason why we should drop the manifest list only constructor.
Contributor
Author
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, with the list-only constructor gone, fromJson now has a single call that passes both manifestList and rootManifest. The v1 embedded-manifests constructor is still separate. |
||
| sequenceNumber, | ||
| snapshotId, | ||
|
|
@@ -188,6 +193,7 @@ static Snapshot fromJson(JsonNode node) { | |
| summary, | ||
| schemaId, | ||
| manifestList, | ||
| rootManifest, | ||
| firstRowId, | ||
| addedRows, | ||
| keyId); | ||
|
|
||
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I followed @gaborkaszab 's suggestion and dropped the list-only overload, keeping a single constructor that takes both locations. Existing callers pass null for rootManifest.
The upside is a single signature. The downside is every previous list-only call site now passes an extra null, which leads to churning in test code.
This is how BaseSnapshot has grown before: schemaId (#2275), firstRowId/addedRows (#11948), and keyId (#12927) were added to the same constructor.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We might want to introduce a dedicated contract once all the other V4 Snapshot fields are introduced, but let it be the concert of the follow-ups, not this one.
Thanks!