Skip to content

Add AWS KMS client and Spark-encrypted table read tests - #4431

Merged
kevinjqliu merged 4 commits into
apache:mainfrom
kevinjqliu:kevinjqliu-encrypted-table-support
Nov 20, 2026
Merged

kevinjqliu merged 4 commits into
apache:mainfrom
kevinjqliu:kevinjqliu-encrypted-table-support

Conversation

@kevinjqliu

@kevinjqliu kevinjqliu commented Nov 19, 2026 •

Copy link
Copy Markdown
Contributor

Rationale for this change

Adds end-to-end tests for encrypted tables. Spark writes encrypted v3 tables, and PyIceberg reads them.

Also adds AwsKeyManagementClient, a KeyManagementClient backed by AWS KMS. Spark's hive catalog and PyIceberg both use AWS KMS, mocked by moto in a new kms service in the integration setup. Each test table gets its own master key.

As of Iceberg Java 1.12, only the Hive catalog supports table encryption (HiveCatalog). REST catalog support is in progress in apache/iceberg#13225.

The tests cover AES-128 and AES-256 data keys:

  • scans, including deletion vectors, filters, and projections
  • time travel
  • missing master key, and Parquet data files that can't be read without their keys
  • inspect metadata tables

Follows the read support in #4159, #4308, #4309, and #4326. Builds on the earlier prototypes in #3427 and #3221 (thanks @smaheshwar-pltr)

Are these changes tested?

Yes.

  • make test-integration: 1372 passed, 71 skipped
  • uv run python -m pytest -m unit tests/encryption/: 161 passed
  • uv run prek run --files dev/docker-compose-integration.yml dev/spark/spark-defaults.conf pyiceberg/encryption/aws_kms.py tests/encryption/test_kms.py tests/integration/test_encryption.py: passed

Are there any user-facing changes?

Yes, adds pyiceberg.encryption.aws_kms.AwsKeyManagementClient. It needs boto3, which isn't part of the encryption extra yet.

kevinjqliu and others added 2 commits November 19, 2026 12:26
Spark writes encrypted v3 tables through its hive catalog with Java's UnitestKMS, and PyIceberg reads them with a matching in-memory KMS.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@kevinjqliu

Copy link
Copy Markdown
Contributor Author

depends on #4199 to pass CI, will rebase once #4199 is merged

Comment thread dev/spark/Dockerfile Resolved
Comment thread dev/spark/spark-defaults.conf Resolved
Comment thread tests/integration/test_encryption.py Resolved

@xanderbailey xanderbailey left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Very cool! This all looks correct and well documented to me!

kevinjqliu and others added 2 commits November 19, 2026 14:23
Adds AwsKeyManagementClient and a moto KMS service, replacing UnitestKMS and the iceberg-core tests jar in the Spark image.

Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
@kevinjqliu kevinjqliu changed the title Add integration tests for reading Spark-encrypted tables Add AWS KMS client and Spark-encrypted table read tests Nov 20, 2026
KMS_ENDPOINT = "kms.endpoint"


class AwsKeyManagementClient(_TransientStateMixin, KeyManagementClient):

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this add an aws kms client.

was using this for testing, but then thought we can just reuse it

Comment on lines +36 to +38
# Table encryption uses AWS KMS, mocked by moto in the kms service
spark.sql.catalog.hive.encryption.kms-type aws
spark.sql.catalog.hive.kms.endpoint http://kms:5001

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

using moto to mock kms service.

only using hive catalog because its the only one supporting table encryption as of Iceberg 1.12

aws_access_key_id="admin",
aws_secret_access_key="password",
)
master_key_id = kms.create_key()["KeyMetadata"]["KeyId"]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

creates a new master_key_id per table

Comment on lines +27 to +28
# AES-128 and AES-256 data keys
DATA_KEY_LENGTHS = [16, 32]

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

parameterize tests by the key lengths

@kevinjqliu
kevinjqliu marked this pull request as ready for review November 20, 2026 03:25
@kevinjqliu
kevinjqliu requested review from Fokko and geruh and a balanced review from Copilot November 20, 2026 03:25
@kevinjqliu

Copy link
Copy Markdown
Contributor Author

@xanderbailey i changed the pr to use aws kms instead, its cleaner than pulling the test jar. Plus now we have a working kms client 😄

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🟢 Approval recommended

No active review issues are identified for the AWS KMS client and encrypted-table read changes.

No open issues 🎉

What changed in this PR

Adds an AWS KMS client and end-to-end coverage for reading Spark-written encrypted v3 tables, building on PyIceberg’s existing encrypted-file read support.

Changes:

  • Adds AwsKeyManagementClient with AWS configuration support and tests for key wrapping, unwrapping, and serialization.
  • Configures a moto-backed KMS service for Spark’s Hive catalog.
  • Tests AES-128 and AES-256 encrypted reads, deletion vectors, filtering, projection, time travel, missing-key failures, and metadata inspection.
File Description
tests/​integration/​test_encryption.py Adds Spark-to-PyIceberg encrypted-table read tests.
tests/​encryption/​test_kms.py Tests AWS KMS round trips and pickling.
pyiceberg/​encryption/​aws_kms.py Implements the AWS KMS key management client.
dev/​spark/​spark-defaults.conf Connects Hive catalog encryption to mock KMS.
dev/​docker-compose-integration.yml Adds the moto KMS integration service.

Review effort: Balanced

@geruh geruh left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@kevinjqliu
kevinjqliu added this pull request to the merge queue Nov 20, 2026
Merged via the queue into apache:main with commit 07216ed Nov 20, 2026
20 checks passed
@kevinjqliu
kevinjqliu deleted the kevinjqliu-encrypted-table-support branch November 20, 2026 04:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants