Skip to content

Core, Spark 4.2: Refactor: Wire in InputFilesDecryptor into Spark readers - #18486

Open
gaborkaszab wants to merge 9 commits into
apache:mainfrom
gaborkaszab:main_spark_readers_InputFiles_decryptor
Open

gaborkaszab wants to merge 9 commits into
apache:mainfrom
gaborkaszab:main_spark_readers_InputFiles_decryptor

Conversation

@gaborkaszab

@gaborkaszab gaborkaszab commented Oct 13, 2026 •

Copy link
Copy Markdown
Contributor

Motivation:

  • Use the same decryptior class across all of the readers: Currently, the readers are asymmetrical, because some of them can read encrypted files, some can't , some can read encrypted data files but not delete files. Wiring in the decryption to all of them will make them able to. Relevant PRs: Flink, GenericReader, IcebergInputFormat. ArrowReader is on my list too
  • Later, extending InputFilesDecryptor to decrypt column files too will be available to all the readers out of the box

Interface change:

  • Now InputFilesDecryptor can handle different types of ScanTasks. Instead of switching on the exact task type, there is a new interface called HasReferencedFiles that is implemented by descendants of ScanTask to provide their referenced files themselves.

@gaborkaszab

Copy link
Copy Markdown
Contributor Author

Thanks for the review, @nastra !
@anuragmantri Do you think you can take a look at this from Spark's perspective?

@anuragmantri anuragmantri left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for working on this, @gaborkaszab. Putting every reader on InputFilesDecryptor makes sense.

What do you think about keeping referencedFiles(TaskT) as the subclass hook and building the decryptor once in BaseReader?

this.inputFilesDecryptor =
    InputFilesDecryptor.fromFiles(
        () -> taskGroup.tasks().stream().flatMap(this::referencedFiles).iterator(), this.fileIO);

There'd be one way to build the decryptor instead of fromTasks in two readers and a hand-built fromFiles in the other two. It also avoids calling the abstract newInputFilesDecryptor() from the BaseReader constructor.

A related question about column files. Where do you plan to add their decryption? If it goes into referencedFiles(Iterable<FileScanTask>), RowDataReader and BatchDataReader would get it, but ChangelogRowReader and PositionDeletesRowReader wouldn't, since they pass their own file lists. Changelog reads of AddedRowsScanTask would need column files too. If it goes in per file inside the decryptor, both shapes get it the same way.

.collect(Collectors.joining(", "));
LOG.error("Error reading file(s): {}", filePaths, e);
if (currentTask instanceof ContentScanTask<?> contentTask && !currentTask.isDataTask()) {
LOG.error("Error reading file: {}", contentTask.file().location(), e);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Before this change a read failure logged every file the task touched, so for a FileScanTask you'd see the data file and its delete files. Now only task.file() shows up. That last one has no path in its message, and it's exactly the case this encryption work targets. Could we keep logging the delete files too? Keeping referencedFiles(TaskT) in BaseReader would let this block stay as it was. I added more detail in the top-level comment.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Logging delete files
    I'm afraid we don't have a unified interface the get the delete files across the implementations: We have a ContentScanTask here that doesn't give an interface for deletes, and I don't think we want to cast to specific types here just for logging.
    I agree that we loose some of the previous functionality here, but I believe that since this is logging, we won't sacrifice any actual functionality by simply logging the data file's location.

  2. referencedFiles()
    One of the main goals of the decryptor call is that once that's wired in, we don;t have to change all the readers to add column files to the referenced files list, we could do that internally in the decryptor making it available to all readers out of the box. I agree that the changeling reader and position delete reader are special in this case constructing their own referenced files list. I made some adjustments to the decryptor to do that internally, moving code out from these readers. WDYT?

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough on the logging, let's drop it.

This is more of a question:

I'd rather not add two public factories that only Spark calls, though. Could we widen fromTasks``it to Iterable<? extends ScanTask>` and branch on the task type inside?

The Flink callers still compile. BaseReader can then call InputFilesDecryptor.fromTasks(taskGroup.tasks(), this.fileIO), and the abstract method, the two accessors and the overrides go away. I tried this on your branch. Core and Spark 4.2 compile, and TestInputFilesDecryptor and TestBaseReader pass. A short Javadoc on fromTasks would be nice too.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was hesitant to do this, but the code seems somewhat simpler this way. Thanks for the suggestion!

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that there's HasReferencedFiles, we can get the old log back cheaply:

if (currentTask instanceof HasReferencedFiles referencing && !currentTask.isDataTask()) {
  String filePaths =
      String.join(
          ", ", Iterables.transform(referencing.referencedFiles(), ContentFile::location));
  LOG.error("Error reading file(s): {}", filePaths, e);
}

I checked that it compiles and passes checkstyle.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, thanks for spotting this!

@gaborkaszab gaborkaszab left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the review, @anuragmantri !

I think your questions are valid, however I wouldn't be much concerned about the logging part. For the referenced files part, see my answer to the inline comment.

@anuragmantri anuragmantri left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the update, @gaborkaszab. I left a few comments on the new decryptor code.

.collect(Collectors.joining(", "));
LOG.error("Error reading file(s): {}", filePaths, e);
if (currentTask instanceof ContentScanTask<?> contentTask && !currentTask.isDataTask()) {
LOG.error("Error reading file: {}", contentTask.file().location(), e);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fair enough on the logging, let's drop it.

This is more of a question:

I'd rather not add two public factories that only Spark calls, though. Could we widen fromTasks``it to Iterable<? extends ScanTask>` and branch on the task type inside?

The Flink callers still compile. BaseReader can then call InputFilesDecryptor.fromTasks(taskGroup.tasks(), this.fileIO), and the abstract method, the two accessors and the overrides go away. I tried this on your branch. Core and Spark 4.2 compile, and TestInputFilesDecryptor and TestBaseReader pass. A short Javadoc on fromTasks would be nice too.

@gaborkaszab gaborkaszab left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for looking into this, @anuragmantri !

.collect(Collectors.joining(", "));
LOG.error("Error reading file(s): {}", filePaths, e);
if (currentTask instanceof ContentScanTask<?> contentTask && !currentTask.isDataTask()) {
LOG.error("Error reading file: {}", contentTask.file().location(), e);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I was hesitant to do this, but the code seems somewhat simpler this way. Thanks for the suggestion!

@github-actions github-actions Bot added the API label Nov 4, 2026

@anuragmantri anuragmantri left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The Spark side looks good to me now. Since the PR changes api/ now, please change the title to API, Core, Spark 4.2: ... and mention the interface in the description. I have some minor comments but this is very close.

*
* @return the content files referenced by this task
*/
Iterable<ContentFile<?>> referencedFiles();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should this include column files once they land? ColumnFile in #17127 isn't a ContentFile, so this would cover data and delete files only.

Also curious whether you considered a default referencedFiles() on ContentScanTask instead. All five implementers already extend it, and it would avoid a new public type.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no common parent of DataFile, DeleteFile and ColumnFile that we could use here, unfortunately. We have a couple of ways to solve this:

  1. In the future if we introduce something like an EncryptedFile we could introduce a variation that returns the common interface, but for now we don't have anything like that. This might be a long-term solution.
  2. We could adapt/wrap a ColumnFile to seem like a ContentFile and then we can return them as part of the list. Might seem a nasty workaround, I probably wouldn't be in favor of this option.
  3. Technically, ContentFile contains the column files. In the decryptor we can get the ColumnFiles if a file is a DataFile instance.

I think I'd prefer 3) because then there is a single place where we change the code for column file coverage and we get it everywhere, where the decryptor is used.

About the other question: I have considered that option too. What I wanted to avoid is to have a default implementation that compiles and runs but then silently misses some files in case of an implementation that doesn't override it. E.g. ContentScanTask returns file(), then DeletedRowsScanTask forgets to override to add addedDeletes(), and existingDeletes().
Also with that, we would need to check if input is an instance of ContentScanTask. Checking for HasReferencedFiles seems more verbose and cleaner to understand what we are doing.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Option 3 makes sense to me, thanks. Agreed on keeping the explicit interface too.

@@ -49,7 +50,7 @@ public InputFilesDecryptor(
}

public static InputFilesDecryptor fromTasks(

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could we add a short Javadoc here? An unsupported task only fails on the first getInputFile call, which isn't obvious from the signature.

Suggested change
public static InputFilesDecryptor fromTasks(
/**
* Returns a decryptor for the files referenced by the given tasks.
*
* <p>Files are resolved on the first call to {@link #getInputFile(String)}, which throws {@link
* IllegalArgumentException} if a task does not implement {@link HasReferencedFiles}.
*/
public static InputFilesDecryptor fromTasks(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh, I always forget to add the javadoc comment to this public method, thanks for pointing that out and for the suggestion.
I applied it, except the exception part because that seemed to much of an implementation detail to me to leak through Javadoc.

.collect(Collectors.joining(", "));
LOG.error("Error reading file(s): {}", filePaths, e);
if (currentTask instanceof ContentScanTask<?> contentTask && !currentTask.isDataTask()) {
LOG.error("Error reading file: {}", contentTask.file().location(), e);

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that there's HasReferencedFiles, we can get the old log back cheaply:

if (currentTask instanceof HasReferencedFiles referencing && !currentTask.isDataTask()) {
  String filePaths =
      String.join(
          ", ", Iterables.transform(referencing.referencedFiles(), ContentFile::location));
  LOG.error("Error reading file(s): {}", filePaths, e);
}

I checked that it compiles and passes checkstyle.

@gaborkaszab gaborkaszab left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the comments once again, @anuragmantri !

*
* @return the content files referenced by this task
*/
Iterable<ContentFile<?>> referencedFiles();

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is no common parent of DataFile, DeleteFile and ColumnFile that we could use here, unfortunately. We have a couple of ways to solve this:

  1. In the future if we introduce something like an EncryptedFile we could introduce a variation that returns the common interface, but for now we don't have anything like that. This might be a long-term solution.
  2. We could adapt/wrap a ColumnFile to seem like a ContentFile and then we can return them as part of the list. Might seem a nasty workaround, I probably wouldn't be in favor of this option.
  3. Technically, ContentFile contains the column files. In the decryptor we can get the ColumnFiles if a file is a DataFile instance.

I think I'd prefer 3) because then there is a single place where we change the code for column file coverage and we get it everywhere, where the decryptor is used.

About the other question: I have considered that option too. What I wanted to avoid is to have a default implementation that compiles and runs but then silently misses some files in case of an implementation that doesn't override it. E.g. ContentScanTask returns file(), then DeletedRowsScanTask forgets to override to add addedDeletes(), and existingDeletes().
Also with that, we would need to check if input is an instance of ContentScanTask. Checking for HasReferencedFiles seems more verbose and cleaner to understand what we are doing.

@@ -49,7 +50,7 @@ public InputFilesDecryptor(
}

public static InputFilesDecryptor fromTasks(

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ahh, I always forget to add the javadoc comment to this public method, thanks for pointing that out and for the suggestion.
I applied it, except the exception part because that seemed to much of an implementation detail to me to leak through Javadoc.

.collect(Collectors.joining(", "));
LOG.error("Error reading file(s): {}", filePaths, e);
if (currentTask instanceof ContentScanTask<?> contentTask && !currentTask.isDataTask()) {
LOG.error("Error reading file: {}", contentTask.file().location(), e);

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, thanks for spotting this!

@pvary

pvary commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

I like the new HasReferencedFiles interface, but this is an api change, so I would like to ask @nastra to take a second look at this too.

Thanks,
Peter

@gaborkaszab gaborkaszab changed the title Core, Spark 4.2: Refactor: Wire in InputFilesDecryptor into Spark readers API, Core, Spark 4.2: Refactor: Wire in InputFilesDecryptor into Spark readers Nov 5, 2026
for (DeleteFile delete : task.deletes()) {
files.put(delete.location(), delete);
for (ScanTask task : tasks) {
if (!(task instanceof HasReferencedFiles referencing)) {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Preconditions.checkArgument

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. However, I think the original version was more readable because the instance check and the cast was merged into the same line.

public interface PositionDeletesScanTask extends ContentScanTask<DeleteFile> {}
public interface PositionDeletesScanTask extends ContentScanTask<DeleteFile>, HasReferencedFiles {
@Override
default Iterable<ContentFile<?>> referencedFiles() {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why not have ContentScanTask extend HasReferencedFiles and have the default impl there?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@anuragmantri asked the same here.
In a nutshell, I considered that too, but I thought that was too error prone: a default implementation in ContentScanTask returning file() could cause wrong results in a child class that forgets to override referencedFiles() but introduces other files too.
The HasReferencedFiles interface seems more self explanatory, and we'd get a clear error if we try to use a ScanTask implementation that doesn't implement the interface.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Now that two of you asked the same. Do you still think it's better to have a default implementation in ContentScanTask? I can change that then.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks for the explanation. I think what you have right now is probably better than having a default impl in ContentScanTask

@nastra

nastra commented Nov 5, 2026

Copy link
Copy Markdown
Contributor

changes still LGTM but I left a few minor comments that would be good to address

@gaborkaszab gaborkaszab left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @nastra !

I addressed your comments. See my answer inline, why I went for not having a default implementation in ContentScanTask.

public interface PositionDeletesScanTask extends ContentScanTask<DeleteFile> {}
public interface PositionDeletesScanTask extends ContentScanTask<DeleteFile>, HasReferencedFiles {
@Override
default Iterable<ContentFile<?>> referencedFiles() {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@anuragmantri asked the same here.
In a nutshell, I considered that too, but I thought that was too error prone: a default implementation in ContentScanTask returning file() could cause wrong results in a child class that forgets to override referencedFiles() but introduces other files too.
The HasReferencedFiles interface seems more self explanatory, and we'd get a clear error if we try to use a ScanTask implementation that doesn't implement the interface.

for (DeleteFile delete : task.deletes()) {
files.put(delete.location(), delete);
for (ScanTask task : tasks) {
if (!(task instanceof HasReferencedFiles referencing)) {

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. However, I think the original version was more readable because the instance check and the cast was merged into the same line.

@anuragmantri anuragmantri left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for iterating on this, @gaborkaszab. LGTM.

Comment on lines +55 to +56
* <p>Resolution of referenced files are performed lazily on the first call of {@link
* #getInputFile(String)}.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
* <p>Resolution of referenced files are performed lazily on the first call of {@link
* #getInputFile(String)}.
* <p>Referenced files are resolved on the first call to {@link #getInputFile(String)}.

*
* @return the content files referenced by this task
*/
Iterable<ContentFile<?>> referencedFiles();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Option 3 makes sense to me, thanks. Agreed on keeping the explicit interface too.

gaborkaszab referenced this pull request in gaborkaszab/iceberg Nov 9, 2026
…ManifestListFile

ManifestListFile and its implementation contains nothing that is specific
to manifest lists. It is more generally related to files that use
TableMetadata.encryptionKeys to store encrypted encryption key metadata
that are referred to by a key ID.
This PR introduces a more general functionality that can be used accross
multiple file types like manifest lists, V4 root manifests, table
statistics and partition statistics. The less general functionality
specific to manifest lists is deprecated or removed where possible.
gaborkaszab referenced this pull request in gaborkaszab/iceberg Nov 9, 2026
…ManifestListFile

ManifestListFile and its implementation contains nothing that is specific
to manifest lists. It is more generally related to files that use
TableMetadata.encryptionKeys to store encrypted encryption key metadata
that are referred to by a key ID.
This PR introduces a more general functionality that can be used accross
multiple file types like manifest lists, V4 root manifests, table
statistics and partition statistics. The less general functionality
specific to manifest lists is deprecated or removed where possible.
package org.apache.iceberg;

/** An object that references content files. */
public interface HasReferencedFiles {

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this needed? The tasks themselves must be supported by engines in order to read, so they can already get the files that need to be decrypted.

I think this should be helpers in Spark, not a new API.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was meant to be part of the refactor to remove the referencedFiles part from Spark. The motivation is that the list of referenced files is coupled with the ScanTask type not with the reader type, and I wanted to avoid each reader (Spark, Flink, Generic, etc.) to implement producing a list of referenced files individually. Practically, we could just pass the ScanTask to the InputFilesDecryptor and since the type of the task determines the list of referenced files, we don't need to rely on each engines gathering them themselves.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why does referencedFiles need to be removed?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Taking a step back, here is my motivation for the refactor: I found that in terms of encryption support, our readers are asymmetric now, e.g. Spark readers can read encrypted data and deletes files, but GenericReader and IcebergInputFormat can't read encrypted delete files, while EqualityConvertReader can't read neither encrypted data nor delete files. The idea is to wire-in InputFilesDecryptor into the readers where we get it guaranteed that given a ScanTask by the reader we have all the referenced files involved with decryption, eliminating asymmetry.
Other motivation is that whenever we add new set of files, e.g. ColumnFiles into the picture, we don't forget to implement encryption support for them. Doing that in InputFilesDecryptor solves this too.

The above proposal makes referencedFiles in Spark redundant with what InputFilesDecryptor is meant to do.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's great to move to a more standard approach, but that doesn't mean this is a reasonable addition to the API. The tasks are public and provide all of this information already. We don't need to add helper methods through some additional interface to the public API.

@gaborkaszab gaborkaszab Nov 10, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The other approach I was exploring did avoid API changes, however had a switch on the ScanTask type inside InputFilesDecryptor. We can move that to a utility class, but I wasn't that comfortable because of the switch. This is how it looked like before, we could move that to a utility or leave it in the decryptor class.
The next step in my thought process was, if we solve the problem at the right level and for me it seemed that the ScanTasks should tell us themselves what files they reference.

What I still think is that the referencedFiles logic should be removed from the readers to make it uniform, regardless if we replace them on a new API or some common utility functionality.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @rdblue !
I removed the API changes and moved the referencedFiles() functionality into an existing utility class in core/ for ScanTasks, TableScanUtil.
Let me know what you think!

@github-actions github-actions Bot removed the API label Nov 11, 2026
@gaborkaszab gaborkaszab changed the title API, Core, Spark 4.2: Refactor: Wire in InputFilesDecryptor into Spark readers Core, Spark 4.2: Refactor: Wire in InputFilesDecryptor into Spark readers Nov 11, 2026
@gaborkaszab
gaborkaszab requested a review from rdblue November 11, 2026 11:39
@gaborkaszab

Copy link
Copy Markdown
Contributor Author

Hey @rdblue ,
The API changes are now replaced with the usage of a utility method in core. Does this address your concerns?

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants