Repository navigation
Core, Spark 4.2: Refactor: Wire in InputFilesDecryptor into Spark readers - #18486
gaborkaszab wants to merge 9 commits into
Conversation
|
Thanks for the review, @nastra ! |
anuragmantri
left a comment
There was a problem hiding this comment.
Thanks for working on this, @gaborkaszab. Putting every reader on InputFilesDecryptor makes sense.
What do you think about keeping referencedFiles(TaskT) as the subclass hook and building the decryptor once in BaseReader?
this.inputFilesDecryptor =
InputFilesDecryptor.fromFiles(
() -> taskGroup.tasks().stream().flatMap(this::referencedFiles).iterator(), this.fileIO);There'd be one way to build the decryptor instead of fromTasks in two readers and a hand-built fromFiles in the other two. It also avoids calling the abstract newInputFilesDecryptor() from the BaseReader constructor.
A related question about column files. Where do you plan to add their decryption? If it goes into referencedFiles(Iterable<FileScanTask>), RowDataReader and BatchDataReader would get it, but ChangelogRowReader and PositionDeletesRowReader wouldn't, since they pass their own file lists. Changelog reads of AddedRowsScanTask would need column files too. If it goes in per file inside the decryptor, both shapes get it the same way.
| .collect(Collectors.joining(", ")); | ||
| LOG.error("Error reading file(s): {}", filePaths, e); | ||
| if (currentTask instanceof ContentScanTask<?> contentTask && !currentTask.isDataTask()) { | ||
| LOG.error("Error reading file: {}", contentTask.file().location(), e); |
There was a problem hiding this comment.
Before this change a read failure logged every file the task touched, so for a FileScanTask you'd see the data file and its delete files. Now only task.file() shows up. That last one has no path in its message, and it's exactly the case this encryption work targets. Could we keep logging the delete files too? Keeping referencedFiles(TaskT) in BaseReader would let this block stay as it was. I added more detail in the top-level comment.
There was a problem hiding this comment.
-
Logging delete files
I'm afraid we don't have a unified interface the get the delete files across the implementations: We have aContentScanTaskhere that doesn't give an interface for deletes, and I don't think we want to cast to specific types here just for logging.
I agree that we loose some of the previous functionality here, but I believe that since this is logging, we won't sacrifice any actual functionality by simply logging the data file's location. -
referencedFiles()
One of the main goals of the decryptor call is that once that's wired in, we don;t have to change all the readers to add column files to the referenced files list, we could do that internally in the decryptor making it available to all readers out of the box. I agree that the changeling reader and position delete reader are special in this case constructing their own referenced files list. I made some adjustments to the decryptor to do that internally, moving code out from these readers. WDYT?
There was a problem hiding this comment.
Fair enough on the logging, let's drop it.
This is more of a question:
I'd rather not add two public factories that only Spark calls, though. Could we widen fromTasks``it to Iterable<? extends ScanTask>` and branch on the task type inside?
The Flink callers still compile. BaseReader can then call InputFilesDecryptor.fromTasks(taskGroup.tasks(), this.fileIO), and the abstract method, the two accessors and the overrides go away. I tried this on your branch. Core and Spark 4.2 compile, and TestInputFilesDecryptor and TestBaseReader pass. A short Javadoc on fromTasks would be nice too.
There was a problem hiding this comment.
I was hesitant to do this, but the code seems somewhat simpler this way. Thanks for the suggestion!
There was a problem hiding this comment.
Now that there's HasReferencedFiles, we can get the old log back cheaply:
if (currentTask instanceof HasReferencedFiles referencing && !currentTask.isDataTask()) {
String filePaths =
String.join(
", ", Iterables.transform(referencing.referencedFiles(), ContentFile::location));
LOG.error("Error reading file(s): {}", filePaths, e);
}I checked that it compiles and passes checkstyle.
There was a problem hiding this comment.
Indeed, thanks for spotting this!
gaborkaszab
left a comment
There was a problem hiding this comment.
Thanks for the review, @anuragmantri !
I think your questions are valid, however I wouldn't be much concerned about the logging part. For the referenced files part, see my answer to the inline comment.
anuragmantri
left a comment
There was a problem hiding this comment.
Thanks for the update, @gaborkaszab. I left a few comments on the new decryptor code.
| .collect(Collectors.joining(", ")); | ||
| LOG.error("Error reading file(s): {}", filePaths, e); | ||
| if (currentTask instanceof ContentScanTask<?> contentTask && !currentTask.isDataTask()) { | ||
| LOG.error("Error reading file: {}", contentTask.file().location(), e); |
There was a problem hiding this comment.
Fair enough on the logging, let's drop it.
This is more of a question:
I'd rather not add two public factories that only Spark calls, though. Could we widen fromTasks``it to Iterable<? extends ScanTask>` and branch on the task type inside?
The Flink callers still compile. BaseReader can then call InputFilesDecryptor.fromTasks(taskGroup.tasks(), this.fileIO), and the abstract method, the two accessors and the overrides go away. I tried this on your branch. Core and Spark 4.2 compile, and TestInputFilesDecryptor and TestBaseReader pass. A short Javadoc on fromTasks would be nice too.
gaborkaszab
left a comment
There was a problem hiding this comment.
Thanks for looking into this, @anuragmantri !
| .collect(Collectors.joining(", ")); | ||
| LOG.error("Error reading file(s): {}", filePaths, e); | ||
| if (currentTask instanceof ContentScanTask<?> contentTask && !currentTask.isDataTask()) { | ||
| LOG.error("Error reading file: {}", contentTask.file().location(), e); |
There was a problem hiding this comment.
I was hesitant to do this, but the code seems somewhat simpler this way. Thanks for the suggestion!
anuragmantri
left a comment
There was a problem hiding this comment.
The Spark side looks good to me now. Since the PR changes api/ now, please change the title to API, Core, Spark 4.2: ... and mention the interface in the description. I have some minor comments but this is very close.
| * | ||
| * @return the content files referenced by this task | ||
| */ | ||
| Iterable<ContentFile<?>> referencedFiles(); |
There was a problem hiding this comment.
Should this include column files once they land? ColumnFile in #17127 isn't a ContentFile, so this would cover data and delete files only.
Also curious whether you considered a default referencedFiles() on ContentScanTask instead. All five implementers already extend it, and it would avoid a new public type.
There was a problem hiding this comment.
There is no common parent of DataFile, DeleteFile and ColumnFile that we could use here, unfortunately. We have a couple of ways to solve this:
- In the future if we introduce something like an
EncryptedFilewe could introduce a variation that returns the common interface, but for now we don't have anything like that. This might be a long-term solution. - We could adapt/wrap a
ColumnFileto seem like aContentFileand then we can return them as part of the list. Might seem a nasty workaround, I probably wouldn't be in favor of this option. - Technically,
ContentFilecontains the column files. In the decryptor we can get theColumnFiles if a file is aDataFileinstance.
I think I'd prefer 3) because then there is a single place where we change the code for column file coverage and we get it everywhere, where the decryptor is used.
About the other question: I have considered that option too. What I wanted to avoid is to have a default implementation that compiles and runs but then silently misses some files in case of an implementation that doesn't override it. E.g. ContentScanTask returns file(), then DeletedRowsScanTask forgets to override to add addedDeletes(), and existingDeletes().
Also with that, we would need to check if input is an instance of ContentScanTask. Checking for HasReferencedFiles seems more verbose and cleaner to understand what we are doing.
There was a problem hiding this comment.
Option 3 makes sense to me, thanks. Agreed on keeping the explicit interface too.
| @@ -49,7 +50,7 @@ public InputFilesDecryptor( | |||
| } | |||
|
|
|||
| public static InputFilesDecryptor fromTasks( | |||
There was a problem hiding this comment.
Could we add a short Javadoc here? An unsupported task only fails on the first getInputFile call, which isn't obvious from the signature.
| public static InputFilesDecryptor fromTasks( | |
| /** | |
| * Returns a decryptor for the files referenced by the given tasks. | |
| * | |
| * <p>Files are resolved on the first call to {@link #getInputFile(String)}, which throws {@link | |
| * IllegalArgumentException} if a task does not implement {@link HasReferencedFiles}. | |
| */ | |
| public static InputFilesDecryptor fromTasks( |
There was a problem hiding this comment.
Ahh, I always forget to add the javadoc comment to this public method, thanks for pointing that out and for the suggestion.
I applied it, except the exception part because that seemed to much of an implementation detail to me to leak through Javadoc.
| .collect(Collectors.joining(", ")); | ||
| LOG.error("Error reading file(s): {}", filePaths, e); | ||
| if (currentTask instanceof ContentScanTask<?> contentTask && !currentTask.isDataTask()) { | ||
| LOG.error("Error reading file: {}", contentTask.file().location(), e); |
There was a problem hiding this comment.
Now that there's HasReferencedFiles, we can get the old log back cheaply:
if (currentTask instanceof HasReferencedFiles referencing && !currentTask.isDataTask()) {
String filePaths =
String.join(
", ", Iterables.transform(referencing.referencedFiles(), ContentFile::location));
LOG.error("Error reading file(s): {}", filePaths, e);
}I checked that it compiles and passes checkstyle.
gaborkaszab
left a comment
There was a problem hiding this comment.
Thanks for the comments once again, @anuragmantri !
| * | ||
| * @return the content files referenced by this task | ||
| */ | ||
| Iterable<ContentFile<?>> referencedFiles(); |
There was a problem hiding this comment.
There is no common parent of DataFile, DeleteFile and ColumnFile that we could use here, unfortunately. We have a couple of ways to solve this:
- In the future if we introduce something like an
EncryptedFilewe could introduce a variation that returns the common interface, but for now we don't have anything like that. This might be a long-term solution. - We could adapt/wrap a
ColumnFileto seem like aContentFileand then we can return them as part of the list. Might seem a nasty workaround, I probably wouldn't be in favor of this option. - Technically,
ContentFilecontains the column files. In the decryptor we can get theColumnFiles if a file is aDataFileinstance.
I think I'd prefer 3) because then there is a single place where we change the code for column file coverage and we get it everywhere, where the decryptor is used.
About the other question: I have considered that option too. What I wanted to avoid is to have a default implementation that compiles and runs but then silently misses some files in case of an implementation that doesn't override it. E.g. ContentScanTask returns file(), then DeletedRowsScanTask forgets to override to add addedDeletes(), and existingDeletes().
Also with that, we would need to check if input is an instance of ContentScanTask. Checking for HasReferencedFiles seems more verbose and cleaner to understand what we are doing.
| @@ -49,7 +50,7 @@ public InputFilesDecryptor( | |||
| } | |||
|
|
|||
| public static InputFilesDecryptor fromTasks( | |||
There was a problem hiding this comment.
Ahh, I always forget to add the javadoc comment to this public method, thanks for pointing that out and for the suggestion.
I applied it, except the exception part because that seemed to much of an implementation detail to me to leak through Javadoc.
| .collect(Collectors.joining(", ")); | ||
| LOG.error("Error reading file(s): {}", filePaths, e); | ||
| if (currentTask instanceof ContentScanTask<?> contentTask && !currentTask.isDataTask()) { | ||
| LOG.error("Error reading file: {}", contentTask.file().location(), e); |
There was a problem hiding this comment.
Indeed, thanks for spotting this!
|
I like the new Thanks, |
| for (DeleteFile delete : task.deletes()) { | ||
| files.put(delete.location(), delete); | ||
| for (ScanTask task : tasks) { | ||
| if (!(task instanceof HasReferencedFiles referencing)) { |
There was a problem hiding this comment.
nit: Preconditions.checkArgument
There was a problem hiding this comment.
Done. However, I think the original version was more readable because the instance check and the cast was merged into the same line.
| public interface PositionDeletesScanTask extends ContentScanTask<DeleteFile> {} | ||
| public interface PositionDeletesScanTask extends ContentScanTask<DeleteFile>, HasReferencedFiles { | ||
| @Override | ||
| default Iterable<ContentFile<?>> referencedFiles() { |
There was a problem hiding this comment.
why not have ContentScanTask extend HasReferencedFiles and have the default impl there?
There was a problem hiding this comment.
@anuragmantri asked the same here.
In a nutshell, I considered that too, but I thought that was too error prone: a default implementation in ContentScanTask returning file() could cause wrong results in a child class that forgets to override referencedFiles() but introduces other files too.
The HasReferencedFiles interface seems more self explanatory, and we'd get a clear error if we try to use a ScanTask implementation that doesn't implement the interface.
There was a problem hiding this comment.
Now that two of you asked the same. Do you still think it's better to have a default implementation in ContentScanTask? I can change that then.
There was a problem hiding this comment.
thanks for the explanation. I think what you have right now is probably better than having a default impl in ContentScanTask
|
changes still LGTM but I left a few minor comments that would be good to address |
gaborkaszab
left a comment
There was a problem hiding this comment.
Thanks @nastra !
I addressed your comments. See my answer inline, why I went for not having a default implementation in ContentScanTask.
| public interface PositionDeletesScanTask extends ContentScanTask<DeleteFile> {} | ||
| public interface PositionDeletesScanTask extends ContentScanTask<DeleteFile>, HasReferencedFiles { | ||
| @Override | ||
| default Iterable<ContentFile<?>> referencedFiles() { |
There was a problem hiding this comment.
@anuragmantri asked the same here.
In a nutshell, I considered that too, but I thought that was too error prone: a default implementation in ContentScanTask returning file() could cause wrong results in a child class that forgets to override referencedFiles() but introduces other files too.
The HasReferencedFiles interface seems more self explanatory, and we'd get a clear error if we try to use a ScanTask implementation that doesn't implement the interface.
| for (DeleteFile delete : task.deletes()) { | ||
| files.put(delete.location(), delete); | ||
| for (ScanTask task : tasks) { | ||
| if (!(task instanceof HasReferencedFiles referencing)) { |
There was a problem hiding this comment.
Done. However, I think the original version was more readable because the instance check and the cast was merged into the same line.
anuragmantri
left a comment
There was a problem hiding this comment.
Thanks for iterating on this, @gaborkaszab. LGTM.
| * <p>Resolution of referenced files are performed lazily on the first call of {@link | ||
| * #getInputFile(String)}. |
There was a problem hiding this comment.
| * <p>Resolution of referenced files are performed lazily on the first call of {@link | |
| * #getInputFile(String)}. | |
| * <p>Referenced files are resolved on the first call to {@link #getInputFile(String)}. |
| * | ||
| * @return the content files referenced by this task | ||
| */ | ||
| Iterable<ContentFile<?>> referencedFiles(); |
There was a problem hiding this comment.
Option 3 makes sense to me, thanks. Agreed on keeping the explicit interface too.
…ManifestListFile ManifestListFile and its implementation contains nothing that is specific to manifest lists. It is more generally related to files that use TableMetadata.encryptionKeys to store encrypted encryption key metadata that are referred to by a key ID. This PR introduces a more general functionality that can be used accross multiple file types like manifest lists, V4 root manifests, table statistics and partition statistics. The less general functionality specific to manifest lists is deprecated or removed where possible.
…ManifestListFile ManifestListFile and its implementation contains nothing that is specific to manifest lists. It is more generally related to files that use TableMetadata.encryptionKeys to store encrypted encryption key metadata that are referred to by a key ID. This PR introduces a more general functionality that can be used accross multiple file types like manifest lists, V4 root manifests, table statistics and partition statistics. The less general functionality specific to manifest lists is deprecated or removed where possible.
| package org.apache.iceberg; | ||
|
|
||
| /** An object that references content files. */ | ||
| public interface HasReferencedFiles { |
There was a problem hiding this comment.
Why is this needed? The tasks themselves must be supported by engines in order to read, so they can already get the files that need to be decrypted.
I think this should be helpers in Spark, not a new API.
There was a problem hiding this comment.
That was meant to be part of the refactor to remove the referencedFiles part from Spark. The motivation is that the list of referenced files is coupled with the ScanTask type not with the reader type, and I wanted to avoid each reader (Spark, Flink, Generic, etc.) to implement producing a list of referenced files individually. Practically, we could just pass the ScanTask to the InputFilesDecryptor and since the type of the task determines the list of referenced files, we don't need to rely on each engines gathering them themselves.
There was a problem hiding this comment.
Why does referencedFiles need to be removed?
There was a problem hiding this comment.
Taking a step back, here is my motivation for the refactor: I found that in terms of encryption support, our readers are asymmetric now, e.g. Spark readers can read encrypted data and deletes files, but GenericReader and IcebergInputFormat can't read encrypted delete files, while EqualityConvertReader can't read neither encrypted data nor delete files. The idea is to wire-in InputFilesDecryptor into the readers where we get it guaranteed that given a ScanTask by the reader we have all the referenced files involved with decryption, eliminating asymmetry.
Other motivation is that whenever we add new set of files, e.g. ColumnFiles into the picture, we don't forget to implement encryption support for them. Doing that in InputFilesDecryptor solves this too.
The above proposal makes referencedFiles in Spark redundant with what InputFilesDecryptor is meant to do.
There was a problem hiding this comment.
It's great to move to a more standard approach, but that doesn't mean this is a reasonable addition to the API. The tasks are public and provide all of this information already. We don't need to add helper methods through some additional interface to the public API.
There was a problem hiding this comment.
The other approach I was exploring did avoid API changes, however had a switch on the ScanTask type inside InputFilesDecryptor. We can move that to a utility class, but I wasn't that comfortable because of the switch. This is how it looked like before, we could move that to a utility or leave it in the decryptor class.
The next step in my thought process was, if we solve the problem at the right level and for me it seemed that the ScanTasks should tell us themselves what files they reference.
What I still think is that the referencedFiles logic should be removed from the readers to make it uniform, regardless if we replace them on a new API or some common utility functionality.
There was a problem hiding this comment.
Thanks @rdblue !
I removed the API changes and moved the referencedFiles() functionality into an existing utility class in core/ for ScanTasks, TableScanUtil.
Let me know what you think!
|
Hey @rdblue , |
Motivation:
ArrowReaderis on my list tooInputFilesDecryptorto decrypt column files too will be available to all the readers out of the boxInterface change:
InputFilesDecryptorcan handle different types ofScanTasks. Instead of switching on the exact task type, there is a new interface calledHasReferencedFilesthat is implemented by descendants ofScanTaskto provide their referenced files themselves.