Repository navigation
Add AWS KMS client and Spark-encrypted table read tests - #4431
Conversation
Spark writes encrypted v3 tables through its hive catalog with Java's UnitestKMS, and PyIceberg reads them with a matching in-memory KMS. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
xanderbailey
left a comment
There was a problem hiding this comment.
Very cool! This all looks correct and well documented to me!
Adds AwsKeyManagementClient and a moto KMS service, replacing UnitestKMS and the iceberg-core tests jar in the Spark image. Co-authored-by: Copilot App <223556219+Copilot@users.noreply.github.com>
| KMS_ENDPOINT = "kms.endpoint" | ||
|
|
||
|
|
||
| class AwsKeyManagementClient(_TransientStateMixin, KeyManagementClient): |
There was a problem hiding this comment.
this add an aws kms client.
was using this for testing, but then thought we can just reuse it
| # Table encryption uses AWS KMS, mocked by moto in the kms service | ||
| spark.sql.catalog.hive.encryption.kms-type aws | ||
| spark.sql.catalog.hive.kms.endpoint http://kms:5001 |
There was a problem hiding this comment.
using moto to mock kms service.
only using hive catalog because its the only one supporting table encryption as of Iceberg 1.12
| aws_access_key_id="admin", | ||
| aws_secret_access_key="password", | ||
| ) | ||
| master_key_id = kms.create_key()["KeyMetadata"]["KeyId"] |
There was a problem hiding this comment.
creates a new master_key_id per table
| # AES-128 and AES-256 data keys | ||
| DATA_KEY_LENGTHS = [16, 32] |
There was a problem hiding this comment.
parameterize tests by the key lengths
|
@xanderbailey i changed the pr to use aws kms instead, its cleaner than pulling the test jar. Plus now we have a working kms client 😄 |
There was a problem hiding this comment.
🟢 Approval recommended
No active review issues are identified for the AWS KMS client and encrypted-table read changes.
No open issues 🎉
What changed in this PR
Adds an AWS KMS client and end-to-end coverage for reading Spark-written encrypted v3 tables, building on PyIceberg’s existing encrypted-file read support.
Changes:
- Adds
AwsKeyManagementClientwith AWS configuration support and tests for key wrapping, unwrapping, and serialization. - Configures a moto-backed KMS service for Spark’s Hive catalog.
- Tests AES-128 and AES-256 encrypted reads, deletion vectors, filtering, projection, time travel, missing-key failures, and metadata inspection.
| File | Description |
|---|---|
tests/integration/test_encryption.py |
Adds Spark-to-PyIceberg encrypted-table read tests. |
tests/encryption/test_kms.py |
Tests AWS KMS round trips and pickling. |
pyiceberg/encryption/aws_kms.py |
Implements the AWS KMS key management client. |
dev/spark/spark-defaults.conf |
Connects Hive catalog encryption to mock KMS. |
dev/docker-compose-integration.yml |
Adds the moto KMS integration service. |
Review effort: Balanced
Rationale for this change
Adds end-to-end tests for encrypted tables. Spark writes encrypted v3 tables, and PyIceberg reads them.
Also adds
AwsKeyManagementClient, aKeyManagementClientbacked by AWS KMS. Spark's hive catalog and PyIceberg both use AWS KMS, mocked by moto in a newkmsservice in the integration setup. Each test table gets its own master key.As of Iceberg Java 1.12, only the Hive catalog supports table encryption (
HiveCatalog). REST catalog support is in progress in apache/iceberg#13225.The tests cover AES-128 and AES-256 data keys:
inspectmetadata tablesFollows the read support in #4159, #4308, #4309, and #4326. Builds on the earlier prototypes in #3427 and #3221 (thanks @smaheshwar-pltr)
Are these changes tested?
Yes.
make test-integration: 1372 passed, 71 skippeduv run python -m pytest -m unit tests/encryption/: 161 passeduv run prek run --files dev/docker-compose-integration.yml dev/spark/spark-defaults.conf pyiceberg/encryption/aws_kms.py tests/encryption/test_kms.py tests/integration/test_encryption.py: passedAre there any user-facing changes?
Yes, adds
pyiceberg.encryption.aws_kms.AwsKeyManagementClient. It needsboto3, which isn't part of theencryptionextra yet.