Skip to content

Core: Plan v4 root-manifest snapshots with the file planner - #18884

Merged
amogh-jahagirdar merged 3 commits into
apache:mainfrom
anoopj:v4-read-path
Nov 13, 2026
Merged

amogh-jahagirdar merged 3 commits into
apache:mainfrom
anoopj:v4-read-path

Conversation

@anoopj

@anoopj anoopj commented Nov 11, 2026 •

Copy link
Copy Markdown
Member

Route a data scan to the native FilePlanner when the selected snapshot references a root manifest

@anoopj
anoopj marked this pull request as draft November 11, 2026 05:42
@anoopj
anoopj force-pushed the v4-read-path branch 2 times, most recently from 2fd96a4 to 344a5e9 Compare November 11, 2026 17:18
@anoopj
anoopj marked this pull request as ready for review November 11, 2026 17:19
@anoopj anoopj moved this to In review in V4: metadata tree Nov 11, 2026

@gaborkaszab gaborkaszab left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @anoopj !
Did an initial pass and left some comments.

@anoopj

anoopj commented Nov 11, 2026

Copy link
Copy Markdown
Member Author

Thanks @anoopj ! Did an initial pass and left some comments.

@gaborkaszab Thanks for the feedback. I was in the middle of cleaning this up, and did a force push. I think my cleanup addressed most of the feedback.

@amogh-jahagirdar amogh-jahagirdar left a comment •

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a test plan I was able to come up with for the scan planning changes. I think it'd fit well in the existing scan suites gated on v4, with commitRootManifestSnapshot moved into V4TestHelpers so both core and Spark can use it. Some of these you already covered, so change needed (mentioned in each row). Let me know what you think!

Test Suite Case
plansRootManifestSnapshot DataTableScanTestBase A scan of a root manifest snapshot plans the files referenced by the root manifest (moved from TestV4ScanPlanning)
rootManifestColumnStatsIgnored TestScanDataFileColumns Tasks don't include column stats unless they're requested
rootManifestColumnStatsLoading TestScanDataFileColumns includeColumnStats() returns stats for all columns
rootManifestColumnStatsPartial TestScanDataFileColumns includeColumnStats(columns) returns stats for only the requested columns
distributedDataPlanningFailsForRootManifest Spark distributed scan DISTRIBUTED data planning mode is rejected for a root manifest snapshot
localDataPlanningPlansRootManifest Spark distributed scan LOCAL data planning mode plans the files referenced by the root manifest
fromResolvesRelativeRootLocation TestRootManifestFile A relative root location resolves against the table location (already there, assert only the resolved path)
fromRejectsEncryptedRoot TestRootManifestFile A snapshot with an encryption key ID is rejected (already there, no change needed)
columnFilesUnsupported TestFilePlanner Planning a file with column files is rejected (already there, no change needed)

String location = LocationUtil.resolveLocation(tableLocation, snapshot.rootManifestLocation());
return new RootManifestFile(
location,
io.newInputFile(location).getLength(),

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

At some point for the encryption work iirc we'd store the root manifest length in the snapshot right? But I agree with making progress here independent of that

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The spec PR for length is still open: #18780 If we merged that, we could add the field similarly to rootManifestLocation done by @stevenzwu recently, and then we could avoid this workaround.
I'm not a fan of introducing such dependencies between PRs, but for me the natural order of progression seems 1) wrap up spec PRs introducing new fields 2) introduce the new fields in the code 3) start using these fields, avoiding workarounding them.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

FileIO is only needed so that we can create an InputFile but that's only needed so that we can get the length (probably by round-tripping to object store). Another reason I think why we should conclude on the spec PR first to add a length field to Snapshot: we could simply remove the FileIO param here and avoid making InputFile getting us a length.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree that we need to make progress independent of the encryption work. I'm skeptical that we are going to want to add length to Snapshot because we added it to key metadata so that it can be encrypted and trusted.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agree this doesn't need to block on the snapshot field, so keeping this PR independent makes sense.

The reason I'd still like the field is that RootManifestFile.from is the only code here that needs FileIO, and it only uses it to fetch the length. Recording the size on the snapshot lets the reader pass the length through newInputFile(path, length) instead of calling getLength(), which avoids a HEAD request on every root read. It also lets RootManifestFile drop the FileIO parameter.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

About the question whether to have a length member in snapshot: There are a number of cases when we don't have the length through key metadata:

  • Encryption is only implemented now for tables in Hive catalog if I'm not mistaken.
  • Parquet files we don't need to save length. We might do in our Java implementation, but not required, so other writers might not write length to Parque files key metadata. Just checked the encryption spec proposal, it also mentions length optional, and not set for Parquet.

I think saving length to Snapshot makes sense.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

A good way to solve this problem is also to make the length lazy. We could pass a Supplier<Long> that calls getLength() if the caller needs it. But we don't want to do that in this PR.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like we have consensus that whether we add a length to snapshot shouldn't block this PR. We will need to determine whether a file length in snapshot is a good idea, and it might be since the size of the root manifest is an important input to writes. But we can decide that later.

return CloseableIterable.transform(dataFiles, this::createTask);
}

private static void rejectColumnFiles(TrackedFile entry) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

cc @rdblue would this check fit better in TrackedFileAdapters.asDataFile, similar to the content type check there? A DataFile is expected to fully describe the file's contents, so any caller treating a file with column files as a DataFile would silently ignore them, not just the planner. Moving it there also keeps the planner as it was, without the two calls and the lambda.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

And then of course once we add the support, we loosen that check. But I'm not sure that planner is the right place for this? Though the coutner argument to that is the scan planning would be responsible for figuring out which files to read based on field IDs.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems more natural to reject such files in the adapter right away. However, rejecting it here gives us the freedom to do development on the reader while we still reject column files in the planner. No strong opinion, though.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1 for adding this to asDataFile since we don't yet support it in DataFile. Good idea, @amogh-jahagirdar!

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we reject this on the DataFile level, we won't be able to make any parallel progress with column files. My immediate follow-up after merging the metadata structs is exposing column files through the tracked file adapters (PR). Without that, all we can do is private experiments.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved it to asDataFile. @gaborkaszab Perhaps your PR can remove the check?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this makes the most sense in asDataFile. DataFile is the public API interface that will expose column files and it currently does not handle column files. When column files are available through that API, we can relocate the check to engines that do not support applying column files. But we can't do that until the API exposes it.

Route a data scan to the v4 `FilePlanner` when the selected snapshot
references a root manifest.
@anoopj

anoopj commented Nov 13, 2026

Copy link
Copy Markdown
Member Author

#17523 got merged. Now it's stacked off main

private final Long firstRowId;
private final byte[] keyMetadata;

RootManifestFile(

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If we introduce a builder method, should this be private?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm keeping the constructor package private for now to avoid the test churn.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving it package-private seems reasonable to me.

class RootManifestFile implements ManifestFile {
private static final int FORMAT_VERSION = 4;

static RootManifestFile from(Snapshot snapshot, FileIO io, String tableLocation) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I wonder what the level of knowledge such a RootManifestFile should contain. For me this seems a simple abstraction that holds a couple of fields like location, length, etc.
However, we now move knowledge about location resolution, FileIO, an entire Snapshot object here, and if I'm not mistaken also encryption related knowledge about how to get keyMetadata from keyId (or from the new fields we potentially add).
I think such a knowledge should live one abstraction level above and RootManifestFile should have a single constructor accepting the pre-computed values. If we have multiple places where we create RootManifestFiles then maybe we should construct them either from Snapshot, or from some class like we have for DataFiles.
WDYT?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We don't need to make this complicated just yet. Right now we need a way to produce a ManifestFile for the root metadata file and this is a simple way to do that. It's also package-private so we can change it later if we need to.

protected CloseableIterable<ScanTask> doPlanFiles() {
Snapshot snapshot = snapshot();

if (snapshot.rootManifestLocation() != null) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like support for distributed scan planning is not required. The Table instance must extend SupportsDistributedScanPlanning, which has a method to enable/disable, allowDistributedPlanning.

That prevents having a BaseDistributedTableScan which means that we don't need to respect dataPlanningMode because the table itself doesn't allow distributed scanning. I think we should go with that easier solution by adding this to BaseTable (between refresh and newScan):

  @Override
  public boolean allowDistributedPlanning() {
    return ops.current().formatVersion() < 4;
  }

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks. This is much simpler and will incorporate it. I didn't know about the table capability.

private int[] statsColumnIds() {
Set<Integer> columns = columnsToKeepStats();
if (columns == null) {
columns = TypeUtil.getProjectedIds(tableSchema());

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This logic is only correct if shouldReturnColumnStats is true.

By default, stats are not copied. There are two ways to produce stats from the scan:

  • Scan.includeColumnStats() returns all stats
  • Scan.includeColumnStats(Collection<Integer>) returns stats for specific field IDs

Both of those methods set shouldReturnColumnStats(true) on the context, but the second one also sets columnsToKeepStats(Set<Integer>). That means columnsToKeepStats() is more specific than returnColumnStats().

The correct logic is:

  • If columnsToKeepStats() is non-null, use that set
  • Otherwise, if returnColumnStats() is true, pass TypeUtil.getProjectedIds(tableSchema())
  • Otherwise, do not call projectStats

That means the correctness of this method is dependent on how it is called. A better solution is to use the Set<Integer> argument as @amogh-jahagirdar suggests and embed this logic above:

    if (shouldReturnColumnStats()) {
      if (columnsToKeepStats() != null) {
        planner.projectStats(columnsToKeepStats());
      } else {
        planner.projectStats(TypeUtil.getProjectedIds(tableSchema()));
      }
    }

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great callout. Thanks! I have incorporated the above logic to newFilePlanner()

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

BTW there was no bug in the old code, because we were doing

if (shouldReturnColumnStats()) {
  planner.projectStats(statsColumnIds());
}

But agree that it reads a bit fragile, so I have inlined it.

}

@Test
void includeColumnStatsReturnsBounds() throws IOException {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think that the purpose of this test is to ensure that the scan's includeColumnStats is passed through correctly. Why only includeColumnStats? We also need to verify:

  • includeColumnStats(Collection<String>)
  • caseSensitive
  • filter
  • ignoreResiduals
  • planWith
  • scanMetrics

Since planFiles is going to return the FilePlanner, you may be able to inspect it instead of running the scan planning.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I also think that these things will be tested when we have v4 tested through existing test suites, so this doesn't necessarily need to add all of these cases. If you're going to have this one, then add the others. But if you want to just verify that this is passing off to FilePlanner in the short term, the suite is fine without this case.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think I will remove this test rather than expanding it.

@anoopj

anoopj commented Nov 13, 2026

Copy link
Copy Markdown
Member Author

This is a test plan I was able to come up with for the scan planning changes. I think it'd fit well in the existing scan suites gated on v4, with commitRootManifestSnapshot moved into V4TestHelpers so both core and Spark can use it. Some of these you already covered, so change needed (mentioned in each row). Let me know what you think!

@amogh-jahagirdar Thanks for writing the test plan. Per the feedback from Ryan, we will keep tests to minimal. The write path will add extensive tests.

@anoopj
anoopj requested a review from rdblue November 13, 2026 21:57
@anoopj
anoopj requested a review from rdblue November 13, 2026 23:02
@amogh-jahagirdar

amogh-jahagirdar commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor

I'm going to go ahead and merge, thank you @anoopj ! We can separately discuss if we want to have a root length per snapshot or other options, and it seems like there was at least consensus that conversation can be done independently of the work being done here to unblock the planning path development.

Thank you @gaborkaszab @rdblue @stevenzwu for reviewing!

@amogh-jahagirdar
amogh-jahagirdar merged commit 38d82b9 into apache:main Nov 13, 2026
40 checks passed
@github-project-automation github-project-automation Bot moved this from In review to Done in V4: metadata tree Nov 13, 2026
@anoopj
anoopj deleted the v4-read-path branch November 14, 2026 01:49
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

5 participants