Skip to content

Data, MR: IcebergInputFormat to read encrypted delete files - #18484

Merged
pvary merged 3 commits into
apache:mainfrom
gaborkaszab:main_IcebergInputFormat_decryption
Nov 18, 2026
Merged

pvary merged 3 commits into
apache:mainfrom
gaborkaszab:main_IcebergInputFormat_decryption

Conversation

@gaborkaszab

Copy link
Copy Markdown
Contributor

Introduces the usage of InputFileDecryptor to IcebergInputFormat to create InputFiles instead of using raw FileIO. Previously this reader failed to read encrypted delete files, but with this change this is fixed.

@gaborkaszab

Copy link
Copy Markdown
Contributor Author

Hi @ggershinsky ,
This is the next step on top of the GenericReader encryption support, extending that to IcebergInputFormat reader in MR. I'd appreciate if you could take a look

@github-actions

Copy link
Copy Markdown

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

@github-actions github-actions Bot added the stale label Nov 16, 2026
@ggershinsky

Copy link
Copy Markdown
Contributor

sorry, somehow I've missed the message

.first();
table.newRowDelta().addDeletes(dv).commit();

assertThat(firstFile.keyMetadata()).isNotNull();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

will this check work if the key metadata is EncryptionKeyMetadata.EMPTY?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ggershinsky In this test we have encrypted files, I'm not sure I see how EncryptionKeyMetadata.EMPTY is relevant here. Isn't that used for non-encrypted tables?
Anyway, with the encrypted table here, the assert seems safe to verify there is a non-null key metadata. If the encryption wouldn't kicked in, the key metadata would be null, see EncryptionKeyMetadata.EMPTY.buffer()` returns null.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM. DataFile.keyMetadata() indeed returns a ByteArray buffer (I wanted to verify this was not a EncryptionKeyMetadata object)

.first();
table.newRowDelta().addDeletes(dv).commit();

assertThat(firstFile.keyMetadata()).isNotNull();

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same question

@pvary pvary added not-stale and removed stale labels Nov 16, 2026

@ggershinsky ggershinsky left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @gaborkaszab

IcebergInputFormat decrypted data files but passed the plain table FileIO
to GenericDeleteFilter, so encrypted delete files were read without
decryption. The record reader now resolves both data and delete files
through an InputFilesDecryptor.

Generated-by: GitHub Copilot CLI (Claude Opus 5.5)
Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
@gaborkaszab
gaborkaszab force-pushed the main_IcebergInputFormat_decryption branch from 994d616 to 14ef121 Compare November 16, 2026 15:02
@gaborkaszab

Copy link
Copy Markdown
Contributor Author

Thanks for the reviews! The GenericReader dependency has been merged, would you mind taking another look at this @nastra @pvary ?

Schema tableSchema,
Schema requestedSchema) {
super(task.file().location(), task.deletes(), tableSchema, requestedSchema);
this.inputFileProvider = inputFileProvider;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nit: Why change this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That was because of how the input param was called was different than the name of the class member. Renamed it to avoid changing this line.

@pvary
pvary merged commit 0d99bcd into apache:main Nov 18, 2026
40 checks passed
@pvary

pvary commented Nov 18, 2026

Copy link
Copy Markdown
Contributor

Merged to main.
Thanks @gaborkaszab for the PR and @ggershinsky and @nastra for the reviews!

@gaborkaszab

Copy link
Copy Markdown
Contributor Author

Thanks for the reviews @ggershinsky @nastra @pvary !

The follow-ups for this is the same for Spark, and I'll prepare ones for ArrowReader and EqualityConvertReader to have full coverage.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants