Repository navigation
Core: Support projecting content stats in the V4 file planner - #18740
Conversation
Add FilePlanner.projectStats(int...) so callers can request content stats for specific table field IDs. The field IDs are forwarded to the root and leaf manifest readers; when unset, no stats are returned. Follow-up to apache#17541 (comment)
amogh-jahagirdar
left a comment
There was a problem hiding this comment.
Overall LGTM, just had some some test comments, thanks @anoopj !
|
|
||
| @ParameterizedTest | ||
| @FieldSource("MANIFEST_FORMATS") | ||
| void columnStatsOmittedByDefault(FileFormat format) throws IOException { |
There was a problem hiding this comment.
Minor: taskOmitsStatsUnlessRequested(or projected instead of requested) reads better imo in terms of the behavior we're trying to express?
There was a problem hiding this comment.
renamed to taskOmitsStatsUnlessProjected. I went with "projected" to match the projectStats(...) API verb.
|
|
||
| @ParameterizedTest | ||
| @FieldSource("MANIFEST_FORMATS") | ||
| void projectStatsForwardedToRootAndLeafReaders(FileFormat format) throws IOException { |
There was a problem hiding this comment.
Similar to above, a comment on the test name: the behavior we're verifying is that tasks come back with stats when they're projected; forwardedToRootAndLeafReader is an implementation detail. Whether a file comes from the root or a leaf doesn't change that behavior, though I agree we should keep covering both. How about tasksContainProjectedStats?
There was a problem hiding this comment.
Renamed to tasksContainProjectedStats. Still keeping one file in the root and one in a leaf so both paths stay covered.
| List<FileScanTask> tasks = | ||
| plan(root, UNPARTITIONED_SPECS, planner -> planner.projectStats(idFieldId)); | ||
|
|
||
| assertThat(tasks) |
There was a problem hiding this comment.
On the assertions: this could be a single extracting/tuple assertion like the one in residualForMixedPartitionedAndUnpartitioned, which keeps both bounds and lets us drop the local paths:
assertThat(tasks)
.extracting(
task -> task.file().location(),
task -> lowerBound(task.file(), idFieldId),
task -> upperBound(task.file(), idFieldId))
.containsExactlyInAnyOrder(tuple(FILE_A, 0, 99), tuple(FILE_C, 100, 199));
Separately, the fixture only has id stats, so this test would also pass if the planner turned on stats for every field instead of the requested ones. Adding data stats to the metadata and asserting they're absent after projection of only id would close that gap.
There was a problem hiding this comment.
Both done. Collapsed to a single tuple-extracting assertion over the shared file constants/.
|
Awesome thanks @anoopj , I'll go ahead and merge since this is a relatively straightforward known follow up change and it looks right to me. |
Add FilePlanner.projectStats(int...) so callers can request content stats for specific table field IDs. The field IDs are forwarded to the root and leaf manifest readers; when unset, no stats are returned.
Follow-up to
#17541 (comment)