Skip to content

Core: Support projecting content stats in the V4 file planner - #18740

Merged
amogh-jahagirdar merged 2 commits into
apache:mainfrom
anoopj:v4-file-planner-project-stats
Nov 2, 2026
Merged

amogh-jahagirdar merged 2 commits into
apache:mainfrom
anoopj:v4-file-planner-project-stats

Conversation

@anoopj

@anoopj anoopj commented Nov 1, 2026

Copy link
Copy Markdown
Member

Add FilePlanner.projectStats(int...) so callers can request content stats for specific table field IDs. The field IDs are forwarded to the root and leaf manifest readers; when unset, no stats are returned.

Follow-up to
#17541 (comment)

Add FilePlanner.projectStats(int...) so callers can request content stats
for specific table field IDs. The field IDs are forwarded to the root and
leaf manifest readers; when unset, no stats are returned.

Follow-up to
apache#17541 (comment)
@github-actions github-actions Bot added the core label Nov 1, 2026
@anoopj anoopj moved this to In review in V4: metadata tree Nov 1, 2026

@amogh-jahagirdar amogh-jahagirdar left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall LGTM, just had some some test comments, thanks @anoopj !


@ParameterizedTest
@FieldSource("MANIFEST_FORMATS")
void columnStatsOmittedByDefault(FileFormat format) throws IOException {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Minor: taskOmitsStatsUnlessRequested(or projected instead of requested) reads better imo in terms of the behavior we're trying to express?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

renamed to taskOmitsStatsUnlessProjected. I went with "projected" to match the projectStats(...) API verb.


@ParameterizedTest
@FieldSource("MANIFEST_FORMATS")
void projectStatsForwardedToRootAndLeafReaders(FileFormat format) throws IOException {

@amogh-jahagirdar amogh-jahagirdar Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Similar to above, a comment on the test name: the behavior we're verifying is that tasks come back with stats when they're projected; forwardedToRootAndLeafReader is an implementation detail. Whether a file comes from the root or a leaf doesn't change that behavior, though I agree we should keep covering both. How about tasksContainProjectedStats?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed to tasksContainProjectedStats. Still keeping one file in the root and one in a leaf so both paths stay covered.

List<FileScanTask> tasks =
plan(root, UNPARTITIONED_SPECS, planner -> planner.projectStats(idFieldId));

assertThat(tasks)

@amogh-jahagirdar amogh-jahagirdar Nov 1, 2026 •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On the assertions: this could be a single extracting/tuple assertion like the one in residualForMixedPartitionedAndUnpartitioned, which keeps both bounds and lets us drop the local paths:


assertThat(tasks)
    .extracting(
        task -> task.file().location(),
        task -> lowerBound(task.file(), idFieldId),
        task -> upperBound(task.file(), idFieldId))
    .containsExactlyInAnyOrder(tuple(FILE_A, 0, 99), tuple(FILE_C, 100, 199));

Separately, the fixture only has id stats, so this test would also pass if the planner turned on stats for every field instead of the requested ones. Adding data stats to the metadata and asserting they're absent after projection of only id would close that gap.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both done. Collapsed to a single tuple-extracting assertion over the shared file constants/.

@amogh-jahagirdar

Copy link
Copy Markdown
Contributor

Awesome thanks @anoopj , I'll go ahead and merge since this is a relatively straightforward known follow up change and it looks right to me.

@amogh-jahagirdar
amogh-jahagirdar merged commit ebf2ff8 into apache:main Nov 2, 2026
40 checks passed
@github-project-automation github-project-automation Bot moved this from In review to Done in V4: metadata tree Nov 2, 2026
@anoopj
anoopj deleted the v4-file-planner-project-stats branch November 2, 2026 16:35
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

2 participants