Repository navigation
Core: Plan v4 root-manifest snapshots with the file planner - #18884
Conversation
2fd96a4 to
344a5e9
Compare
gaborkaszab
left a comment
There was a problem hiding this comment.
Thanks @anoopj !
Did an initial pass and left some comments.
@gaborkaszab Thanks for the feedback. I was in the middle of cleaning this up, and did a force push. I think my cleanup addressed most of the feedback. |
There was a problem hiding this comment.
This is a test plan I was able to come up with for the scan planning changes. I think it'd fit well in the existing scan suites gated on v4, with commitRootManifestSnapshot moved into V4TestHelpers so both core and Spark can use it. Some of these you already covered, so change needed (mentioned in each row). Let me know what you think!
| Test | Suite | Case |
|---|---|---|
plansRootManifestSnapshot |
DataTableScanTestBase |
A scan of a root manifest snapshot plans the files referenced by the root manifest (moved from TestV4ScanPlanning) |
rootManifestColumnStatsIgnored |
TestScanDataFileColumns |
Tasks don't include column stats unless they're requested |
rootManifestColumnStatsLoading |
TestScanDataFileColumns |
includeColumnStats() returns stats for all columns |
rootManifestColumnStatsPartial |
TestScanDataFileColumns |
includeColumnStats(columns) returns stats for only the requested columns |
distributedDataPlanningFailsForRootManifest |
Spark distributed scan | DISTRIBUTED data planning mode is rejected for a root manifest snapshot |
localDataPlanningPlansRootManifest |
Spark distributed scan | LOCAL data planning mode plans the files referenced by the root manifest |
fromResolvesRelativeRootLocation |
TestRootManifestFile |
A relative root location resolves against the table location (already there, assert only the resolved path) |
fromRejectsEncryptedRoot |
TestRootManifestFile |
A snapshot with an encryption key ID is rejected (already there, no change needed) |
columnFilesUnsupported |
TestFilePlanner |
Planning a file with column files is rejected (already there, no change needed) |
| String location = LocationUtil.resolveLocation(tableLocation, snapshot.rootManifestLocation()); | ||
| return new RootManifestFile( | ||
| location, | ||
| io.newInputFile(location).getLength(), |
There was a problem hiding this comment.
At some point for the encryption work iirc we'd store the root manifest length in the snapshot right? But I agree with making progress here independent of that
There was a problem hiding this comment.
The spec PR for length is still open: #18780 If we merged that, we could add the field similarly to rootManifestLocation done by @stevenzwu recently, and then we could avoid this workaround.
I'm not a fan of introducing such dependencies between PRs, but for me the natural order of progression seems 1) wrap up spec PRs introducing new fields 2) introduce the new fields in the code 3) start using these fields, avoiding workarounding them.
There was a problem hiding this comment.
FileIO is only needed so that we can create an InputFile but that's only needed so that we can get the length (probably by round-tripping to object store). Another reason I think why we should conclude on the spec PR first to add a length field to Snapshot: we could simply remove the FileIO param here and avoid making InputFile getting us a length.
There was a problem hiding this comment.
I agree that we need to make progress independent of the encryption work. I'm skeptical that we are going to want to add length to Snapshot because we added it to key metadata so that it can be encrypted and trusted.
There was a problem hiding this comment.
Agree this doesn't need to block on the snapshot field, so keeping this PR independent makes sense.
The reason I'd still like the field is that RootManifestFile.from is the only code here that needs FileIO, and it only uses it to fetch the length. Recording the size on the snapshot lets the reader pass the length through newInputFile(path, length) instead of calling getLength(), which avoids a HEAD request on every root read. It also lets RootManifestFile drop the FileIO parameter.
There was a problem hiding this comment.
About the question whether to have a length member in snapshot: There are a number of cases when we don't have the length through key metadata:
- Encryption is only implemented now for tables in Hive catalog if I'm not mistaken.
- Parquet files we don't need to save length. We might do in our Java implementation, but not required, so other writers might not write length to Parque files key metadata. Just checked the encryption spec proposal, it also mentions length optional, and not set for Parquet.
I think saving length to Snapshot makes sense.
There was a problem hiding this comment.
A good way to solve this problem is also to make the length lazy. We could pass a Supplier<Long> that calls getLength() if the caller needs it. But we don't want to do that in this PR.
There was a problem hiding this comment.
Looks like we have consensus that whether we add a length to snapshot shouldn't block this PR. We will need to determine whether a file length in snapshot is a good idea, and it might be since the size of the root manifest is an important input to writes. But we can decide that later.
| return CloseableIterable.transform(dataFiles, this::createTask); | ||
| } | ||
|
|
||
| private static void rejectColumnFiles(TrackedFile entry) { |
There was a problem hiding this comment.
cc @rdblue would this check fit better in TrackedFileAdapters.asDataFile, similar to the content type check there? A DataFile is expected to fully describe the file's contents, so any caller treating a file with column files as a DataFile would silently ignore them, not just the planner. Moving it there also keeps the planner as it was, without the two calls and the lambda.
There was a problem hiding this comment.
And then of course once we add the support, we loosen that check. But I'm not sure that planner is the right place for this? Though the coutner argument to that is the scan planning would be responsible for figuring out which files to read based on field IDs.
There was a problem hiding this comment.
Seems more natural to reject such files in the adapter right away. However, rejecting it here gives us the freedom to do development on the reader while we still reject column files in the planner. No strong opinion, though.
There was a problem hiding this comment.
+1 for adding this to asDataFile since we don't yet support it in DataFile. Good idea, @amogh-jahagirdar!
There was a problem hiding this comment.
If we reject this on the DataFile level, we won't be able to make any parallel progress with column files. My immediate follow-up after merging the metadata structs is exposing column files through the tracked file adapters (PR). Without that, all we can do is private experiments.
There was a problem hiding this comment.
Moved it to asDataFile. @gaborkaszab Perhaps your PR can remove the check?
There was a problem hiding this comment.
I think this makes the most sense in asDataFile. DataFile is the public API interface that will expose column files and it currently does not handle column files. When column files are available through that API, we can relocate the check to engines that do not support applying column files. But we can't do that until the API exposes it.
Route a data scan to the v4 `FilePlanner` when the selected snapshot references a root manifest.
|
#17523 got merged. Now it's stacked off |
| private final Long firstRowId; | ||
| private final byte[] keyMetadata; | ||
|
|
||
| RootManifestFile( |
There was a problem hiding this comment.
If we introduce a builder method, should this be private?
There was a problem hiding this comment.
I'm keeping the constructor package private for now to avoid the test churn.
There was a problem hiding this comment.
Leaving it package-private seems reasonable to me.
| class RootManifestFile implements ManifestFile { | ||
| private static final int FORMAT_VERSION = 4; | ||
|
|
||
| static RootManifestFile from(Snapshot snapshot, FileIO io, String tableLocation) { |
There was a problem hiding this comment.
I wonder what the level of knowledge such a RootManifestFile should contain. For me this seems a simple abstraction that holds a couple of fields like location, length, etc.
However, we now move knowledge about location resolution, FileIO, an entire Snapshot object here, and if I'm not mistaken also encryption related knowledge about how to get keyMetadata from keyId (or from the new fields we potentially add).
I think such a knowledge should live one abstraction level above and RootManifestFile should have a single constructor accepting the pre-computed values. If we have multiple places where we create RootManifestFiles then maybe we should construct them either from Snapshot, or from some class like we have for DataFiles.
WDYT?
There was a problem hiding this comment.
We don't need to make this complicated just yet. Right now we need a way to produce a ManifestFile for the root metadata file and this is a simple way to do that. It's also package-private so we can change it later if we need to.
| protected CloseableIterable<ScanTask> doPlanFiles() { | ||
| Snapshot snapshot = snapshot(); | ||
|
|
||
| if (snapshot.rootManifestLocation() != null) { |
There was a problem hiding this comment.
It looks like support for distributed scan planning is not required. The Table instance must extend SupportsDistributedScanPlanning, which has a method to enable/disable, allowDistributedPlanning.
That prevents having a BaseDistributedTableScan which means that we don't need to respect dataPlanningMode because the table itself doesn't allow distributed scanning. I think we should go with that easier solution by adding this to BaseTable (between refresh and newScan):
@Override
public boolean allowDistributedPlanning() {
return ops.current().formatVersion() < 4;
}There was a problem hiding this comment.
Thanks. This is much simpler and will incorporate it. I didn't know about the table capability.
| private int[] statsColumnIds() { | ||
| Set<Integer> columns = columnsToKeepStats(); | ||
| if (columns == null) { | ||
| columns = TypeUtil.getProjectedIds(tableSchema()); |
There was a problem hiding this comment.
This logic is only correct if shouldReturnColumnStats is true.
By default, stats are not copied. There are two ways to produce stats from the scan:
Scan.includeColumnStats()returns all statsScan.includeColumnStats(Collection<Integer>)returns stats for specific field IDs
Both of those methods set shouldReturnColumnStats(true) on the context, but the second one also sets columnsToKeepStats(Set<Integer>). That means columnsToKeepStats() is more specific than returnColumnStats().
The correct logic is:
- If
columnsToKeepStats()is non-null, use that set - Otherwise, if
returnColumnStats()is true, passTypeUtil.getProjectedIds(tableSchema()) - Otherwise, do not call
projectStats
That means the correctness of this method is dependent on how it is called. A better solution is to use the Set<Integer> argument as @amogh-jahagirdar suggests and embed this logic above:
if (shouldReturnColumnStats()) {
if (columnsToKeepStats() != null) {
planner.projectStats(columnsToKeepStats());
} else {
planner.projectStats(TypeUtil.getProjectedIds(tableSchema()));
}
}There was a problem hiding this comment.
Great callout. Thanks! I have incorporated the above logic to newFilePlanner()
There was a problem hiding this comment.
BTW there was no bug in the old code, because we were doing
if (shouldReturnColumnStats()) {
planner.projectStats(statsColumnIds());
}But agree that it reads a bit fragile, so I have inlined it.
| } | ||
|
|
||
| @Test | ||
| void includeColumnStatsReturnsBounds() throws IOException { |
There was a problem hiding this comment.
I think that the purpose of this test is to ensure that the scan's includeColumnStats is passed through correctly. Why only includeColumnStats? We also need to verify:
includeColumnStats(Collection<String>)caseSensitivefilterignoreResidualsplanWithscanMetrics
Since planFiles is going to return the FilePlanner, you may be able to inspect it instead of running the scan planning.
There was a problem hiding this comment.
I also think that these things will be tested when we have v4 tested through existing test suites, so this doesn't necessarily need to add all of these cases. If you're going to have this one, then add the others. But if you want to just verify that this is passing off to FilePlanner in the short term, the suite is fine without this case.
There was a problem hiding this comment.
I think I will remove this test rather than expanding it.
@amogh-jahagirdar Thanks for writing the test plan. Per the feedback from Ryan, we will keep tests to minimal. The write path will add extensive tests. |
|
I'm going to go ahead and merge, thank you @anoopj ! We can separately discuss if we want to have a root length per snapshot or other options, and it seems like there was at least consensus that conversation can be done independently of the work being done here to unblock the planning path development. Thank you @gaborkaszab @rdblue @stevenzwu for reviewing! |
Route a data scan to the native FilePlanner when the selected snapshot references a root manifest