Repository navigation
Data, MR: IcebergInputFormat to read encrypted delete files - #18484
Conversation
84bda7c to
994d616
Compare
|
Hi @ggershinsky , |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
sorry, somehow I've missed the message |
| .first(); | ||
| table.newRowDelta().addDeletes(dv).commit(); | ||
|
|
||
| assertThat(firstFile.keyMetadata()).isNotNull(); |
There was a problem hiding this comment.
will this check work if the key metadata is EncryptionKeyMetadata.EMPTY?
There was a problem hiding this comment.
@ggershinsky In this test we have encrypted files, I'm not sure I see how EncryptionKeyMetadata.EMPTY is relevant here. Isn't that used for non-encrypted tables?
Anyway, with the encrypted table here, the assert seems safe to verify there is a non-null key metadata. If the encryption wouldn't kicked in, the key metadata would be null, see EncryptionKeyMetadata.EMPTY.buffer()` returns null.
There was a problem hiding this comment.
SGTM. DataFile.keyMetadata() indeed returns a ByteArray buffer (I wanted to verify this was not a EncryptionKeyMetadata object)
| .first(); | ||
| table.newRowDelta().addDeletes(dv).commit(); | ||
|
|
||
| assertThat(firstFile.keyMetadata()).isNotNull(); |
IcebergInputFormat decrypted data files but passed the plain table FileIO to GenericDeleteFilter, so encrypted delete files were read without decryption. The record reader now resolves both data and delete files through an InputFilesDecryptor. Generated-by: GitHub Copilot CLI (Claude Opus 5.5) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
994d616 to
14ef121
Compare
| Schema tableSchema, | ||
| Schema requestedSchema) { | ||
| super(task.file().location(), task.deletes(), tableSchema, requestedSchema); | ||
| this.inputFileProvider = inputFileProvider; |
There was a problem hiding this comment.
That was because of how the input param was called was different than the name of the class member. Renamed it to avoid changing this line.
|
Merged to main. |
|
Thanks for the reviews @ggershinsky @nastra @pvary ! The follow-ups for this is the same for Spark, and I'll prepare ones for |
Introduces the usage of
InputFileDecryptortoIcebergInputFormatto createInputFiles instead of using rawFileIO. Previously this reader failed to read encrypted delete files, but with this change this is fixed.