Repository navigation
Spec: Document standard encryption key metadata - #16527
xanderbailey wants to merge 8 commits into
Conversation
|
cc: @ggershinsky in case you have any thoughts here |
57ab53b to
d67101b
Compare
…ementation interop Specify the binary wire format for the key_metadata field in manifest entries and the key hierarchy in table metadata encryption-keys, which were previously described only as "implementation-specific" but are required for implementations to interoperate.
d67101b to
ae07984
Compare
szlta
left a comment
There was a problem hiding this comment.
Looks good to me, just a minor nits.
|
|
||
| The standard encryption scheme uses a two-tier key hierarchy tracked in the table metadata `encryption-keys` list: | ||
|
|
||
| 1. **Key Encryption Keys (KEKs):** Entries where `encrypted-by-id` equals the table's encryption key ID (configured via `encryption.key-id`). The `encrypted-key-metadata` contains the KEK wrapped by the KMS and is opaque to Iceberg — its format is determined by the KMS provider. |
There was a problem hiding this comment.
Perhaps mention here that a KEY_TIMESTAMP property is expected to be present for KEKs - AFAIK without it decryption flow will error out.
| | Field name | Avro type | Required | Description | | ||
| |---|---|---|---| | ||
| | **`encryption_key`** | `bytes` | _required_ | The data encryption key (DEK) for this file. Must be 16, 24, or 32 bytes (corresponding to AES-128, AES-192, or AES-256). | | ||
| | **`aad_prefix`** | `bytes` | _optional_ | Random AAD prefix used for [AES GCM Stream](gcm-stream-spec.md) block authentication. | |
There was a problem hiding this comment.
| | **`aad_prefix`** | `bytes` | _optional_ | Random AAD prefix used for [AES GCM Stream](gcm-stream-spec.md) block authentication. | | |
| | **`aad_prefix`** | `bytes` | _optional_ | Random AAD prefix used for [AES GCM Stream](gcm-stream-spec.md) integrity protection. | |
|
I think this makes a lot of sense, thank you. I'll have a look at the details. |
- Clarify that aad_prefix is for integrity protection, not authentication - Document that KEY_TIMESTAMP property is required on KEK entries
| | _optional_ | _optional_ | _optional_ | **`125 lower_bounds`** | `map<126: int, 127: binary>` | Map from column id to lower bound in the column serialized as binary [1]. Each value must be less than or equal to all non-null, non-NaN values in the column for the file [2] | | ||
| | _optional_ | _optional_ | _optional_ | **`128 upper_bounds`** | `map<129: int, 130: binary>` | Map from column id to upper bound in the column serialized as binary [1]. Each value must be greater than or equal to all non-null, non-Nan values in the column for the file [2] | | ||
| | _optional_ | _optional_ | _optional_ | **`131 key_metadata`** | `binary` | Implementation-specific key metadata for encryption | | ||
| | _optional_ | _optional_ | _optional_ | **`131 key_metadata`** | `binary` | Per-file encryption key metadata. See [Standard Key Metadata](#standard-key-metadata) for the interoperable format used by the standard encryption scheme. | |
There was a problem hiding this comment.
there is also a key_metadata field in the Manifest File struct (field id 519)
| | Field name | Avro type | Required | Description | | ||
| |---|---|---|---| | ||
| | **`encryption_key`** | `bytes` | _required_ | The data encryption key (DEK) for this file. Must be 16, 24, or 32 bytes (corresponding to AES-128, AES-192, or AES-256). | | ||
| | **`aad_prefix`** | `bytes` | _optional_ | Random AAD prefix used for [AES GCM Stream](gcm-stream-spec.md) integrity protection. | |
There was a problem hiding this comment.
AAD prefix is used not only in AES GCM Stream files, but also in encrypted Parquet files (https://parquet.apache.org/docs/file-format/data-pages/encryption/ or https://github.com/apache/parquet-format/blob/master/Encryption.md)
| | **`aad_prefix`** | `bytes` | _optional_ | Random AAD prefix used for [AES GCM Stream](gcm-stream-spec.md) integrity protection. | | ||
| | **`file_length`** | `long` | _optional_ | The plaintext file length before encryption. Used to detect truncation attacks (see [AES GCM Stream file length](gcm-stream-spec.md#file-length)). | | ||
|
|
||
| The AAD prefix is combined with a 4-byte little-endian block index to form the AAD for each AES GCM Stream cipher block, as described in the [AES GCM Stream AAD section](gcm-stream-spec.md#additional-authenticated-data). |
There was a problem hiding this comment.
in Parquet encryption, this works differently, https://parquet.apache.org/docs/file-format/data-pages/encryption/
| |---|---|---|---| | ||
| | **`encryption_key`** | `bytes` | _required_ | The data encryption key (DEK) for this file. Must be 16, 24, or 32 bytes (corresponding to AES-128, AES-192, or AES-256). | | ||
| | **`aad_prefix`** | `bytes` | _optional_ | Random AAD prefix used for [AES GCM Stream](gcm-stream-spec.md) integrity protection. | | ||
| | **`file_length`** | `long` | _optional_ | The plaintext file length before encryption. Used to detect truncation attacks (see [AES GCM Stream file length](gcm-stream-spec.md#file-length)). | |
There was a problem hiding this comment.
This keeps file length after encryption, https://github.com/apache/iceberg/blob/main/core/src/main/java/org/apache/iceberg/encryption/AesGcmInputFile.java#L45
Only for AES GCM Stream files. Not set/used for encrypted Parquet data files.
b58d594 to
7d60b4c
Compare
Clarify that file_length stores the encrypted (ciphertext) length, not plaintext. Broaden aad_prefix and file_length descriptions to cover both AES GCM Stream and Parquet modular encryption formats.
4690c89 to
4f12e28
Compare
|
Have hopefully addressed comments! Thanks folks. What's the process for getting spec changes merged? Do people typically vote or is just a regular PR approval? |
| | Field name | Avro type | Required | Description | | ||
| |---|---|---|---| | ||
| | **`encryption_key`** | `bytes` | _required_ | The data encryption key (DEK) for this file. Must be 16, 24, or 32 bytes (corresponding to AES-128, AES-192, or AES-256). | | ||
| | **`aad_prefix`** | `bytes` | _optional_ | Random AAD prefix used for encryption integrity protection. For [AES GCM Stream](gcm-stream-spec.md) files, the prefix is combined with a block index to form the per-block AAD. For [Parquet modular encryption](https://parquet.apache.org/docs/file-format/data-pages/encryption/), the prefix is passed as the `aad_file_unique` component. | |
There was a problem hiding this comment.
For Parquet, it also passed as an AAD Prefix parameter; it then combined with AAD Suffixes.
aad_file_unique is something else (a Parquet-internal parameter)
|
|
||
| The usage of the `encryption_key` and `aad_prefix` fields depends on the file format: | ||
|
|
||
| * **AES GCM Stream files** (manifest lists, manifests, and non-Parquet data files): The `encryption_key` is used directly as the AES-GCM key. The `aad_prefix` is combined with a 4-byte little-endian block index to form the AAD for each cipher block, as described in the [AES GCM Stream AAD section](gcm-stream-spec.md#additional-authenticated-data). The `file_length` field stores the encrypted file length for truncation detection. |
There was a problem hiding this comment.
Iceberg table encryption is not supported yet for ORC data format, so "manifest lists, manifests, and non-Parquet data files" should be something like "manifest lists, manifests, and Avro data files"
There was a problem hiding this comment.
Actually delete vectors (Puffin files) are also AES GCM encrypted for encrypted tables. I think the list is long enough to worth showing it as a proper list in the spec rather than in just brackets?
4fce858 to
bcaf862
Compare
| | Field name | Avro type | Required | Description | | ||
| |---|---|---|---| | ||
| | **`encryption_key`** | `bytes` | _required_ | The data encryption key (DEK) for this file. Must be 16, 24, or 32 bytes (corresponding to AES-128, AES-192, or AES-256). | | ||
| | **`aad_prefix`** | `bytes` | _optional_ | Random AAD prefix used for encryption integrity protection. For [AES GCM Stream](gcm-stream-spec.md) files, the prefix is combined with a block index to form the per-block AAD. For [Parquet modular encryption](https://parquet.apache.org/docs/file-format/data-pages/encryption/), the prefix is passed as the AAD prefix parameter, which is combined with a per-file random `aad_file_unique` to form the full file AAD. | |
There was a problem hiding this comment.
I'd suggest changing "the prefix is passed as the AAD prefix parameter, which is combined with a per-file random aad_file_unique to form the full file AAD." to something like "the prefix is passed as the AAD prefix parameter, which is combined with a module AAD suffix to form the full AAD for each Parquet module."
(per the PME spec, aad_file_unique is a part of the suffix)
|
I don't think that this should go into the table spec. The intent when we started work on encryption was to keep the table parts generic and write an encryption-specific spec. I think an encryption spec is the right path forward. |
|
Thanks for taking a look @rdblue! Apologies if I haven't understood correctly, is your suggestion to add a new file in addition to the gcm stream encryption file? And reference it from the main spec? |
|
Hi @rdblue, any guidance here on how to move forward? Happy to change things up here I'm just not sure on the next steps. |
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
Please don't close, not stale. |
|
Gentle nudge here @rdblue for some guidance on what the next steps here should be. Thanks in advance! |
|
I will try to raise this on the July 15 community call if we don't have some movement here by then. |
2bd9086 to
21b7a21
Compare
xanderbailey
left a comment
There was a problem hiding this comment.
Wonder if we need to bring some of the https://github.com/apache/iceberg/blob/main/docs/docs/encryption.md into this spec also. Interested to hear people's thoughts.
|
This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions. |
|
@rdblue whats the best way for me to get a review here? |
singhpk234
left a comment
There was a problem hiding this comment.
Thanks @xanderbailey, i am supportive of documenting this !
i added some questions inline it more intended to folks interested / working group of encryption not just specific to the pr
| where: | ||
|
|
||
| * `VersionByte` is a single byte indicating the key metadata schema version. Currently, the only valid version is `0x01`. | ||
| * `Payload` is an Avro binary-encoded record (not a container file — only the raw binary encoding of the fields) using the schema for the given version. |
There was a problem hiding this comment.
[doubt] how are other language implementations dealing with it ? they are using avro library, or this encoding are standard enough to be replicated ?
There was a problem hiding this comment.
This is the rust implementation and it works correctly.
| * **AES GCM Stream files**: | ||
| - Manifest lists | ||
| - Manifests | ||
| - Avro data files |
There was a problem hiding this comment.
what about ORC ?
is this abstracted carefully with the FileFormat API ?
There was a problem hiding this comment.
ORC has it's own encryption spec I believe and doesn't stream encrypt https://orc.apache.org/specification/ORCv1/
|
|
||
| The standard encryption scheme uses a two-tier key hierarchy tracked in the table metadata [`encryption-keys`](spec.md#encryption-keys) list: | ||
|
|
||
| 1. **Key Encryption Keys (KEKs):** Entries where `encrypted-by-id` equals the table's encryption key ID (configured via `encryption.key-id`). The `encrypted-key-metadata` contains the KEK wrapped by the KMS and is opaque to Iceberg — its format is determined by the KMS provider. KEK entries must include a `KEY_TIMESTAMP` property recording the creation time in milliseconds since epoch; this timestamp is used as the AAD when encrypting manifest list key metadata. |
There was a problem hiding this comment.
[doubt] why is KEK_TIMESTAMP hard coded expiry 720 days ? i know its the maximal time, i wonder if this is something we should make configurable, systems would prefer monthly rotations ?
There was a problem hiding this comment.
It's constant today in java
There was a problem hiding this comment.
I think making it configurable is fine. But it does not affect this spec patch.
There was a problem hiding this comment.
cool its not for this spec but i wonder if this is something we should document, how do you feel clients having different KEK lifespan_ms for now ?
There was a problem hiding this comment.
can be workable, I think. But if this made say a table property, then all clients will use the same lifespan.
I think that would be valuable as different client libraries start to implement this. Might be worth a mailing list thread? |
singhpk234
left a comment
There was a problem hiding this comment.
This mostly looks good to me !
if @ggershinsky is fine lets move ahead to VOTE, there seems to general concencus of adding this
| ### Encryption Key Hierarchy | ||
|
|
||
| The standard encryption scheme uses a two-tier key hierarchy tracked in the table metadata [`encryption-keys`](spec.md#encryption-keys) list: | ||
|
|
||
| 1. **Key Encryption Keys (KEKs):** Entries where `encrypted-by-id` equals the table's encryption key ID (configured via `encryption.key-id`). The `encrypted-key-metadata` contains the KEK wrapped by the KMS and is opaque to Iceberg — its format is determined by the KMS provider. KEK entries must include a `KEY_TIMESTAMP` property recording the creation time in milliseconds since epoch; this timestamp is used as the AAD when encrypting manifest list key metadata. | ||
|
|
||
| 2. **Manifest List Keys:** Entries where `encrypted-by-id` references a KEK. The `encrypted-key-metadata` contains the Standard Key Metadata (defined above) encrypted with AES GCM using the referenced unwrapped KEK. The ciphertext format is: | ||
|
|
||
| ``` | ||
| Nonce Ciphertext Tag | ||
| ``` | ||
|
|
||
| where `Nonce` is 12 bytes, `Ciphertext` is the encrypted Standard Key Metadata payload, and `Tag` is the 16-byte GCM authentication tag. The AAD for this encryption is the KEK's `KEY_TIMESTAMP` property value encoded as UTF-8 bytes. | ||
|
|
||
| The snapshot field `key-id` references the encryption key entry used to encrypt that snapshot's manifest list key metadata. |
There was a problem hiding this comment.
[optional] I understand the text ! i wonder if we should have some sort of a DAG diagram of these key hierrarchy ?
KEKs can be shared by snapshots and is encrypted by TMK, MLK is encrypted by KEK .... .
There was a problem hiding this comment.
Yes, I think the intention is that KEK is used to encrypt multiple manifest list keys (this doesn't have to be the case by the spec I think and implementations would work with multiple KEK - MLKs). I don't think this needs to be enforced by the spec. WDYT?
There was a problem hiding this comment.
I agree i was saying it just from visualization pov, its just an optional comment !
|
|
||
| * **AES GCM Stream files**: | ||
| - Manifest lists | ||
| - Manifests |
There was a problem hiding this comment.
[orthogonal] is there a plan to utilize the PME when in v4 manifests are written in parquet ? if yes how
There was a problem hiding this comment.
I'm not aware of a conversation about this but would be happy to start a thread in the mailing list.
126b741 to
b3756cd
Compare
|
VOTE thread here https://lists.apache.org/thread/hpdmzppsl87l0olkpzm3q47occ9cfydo |
gaborkaszab
left a comment
There was a problem hiding this comment.
I believe we should reduce the coupling between the table spec and the encryption spec. I feel that the current version talks too much about the exact table metadata elements where each encryption technique is meant to be used, and I think rather that we could be more generic with the encryption spec.
|
|
||
| Encrypted key material is tracked in two places: | ||
|
|
||
| * The `key_metadata` fields in [manifest entries](spec.md#manifests), [manifest list entries](spec.md#manifest-lists), and [statistics files](spec.md#table-metadata-fields) store the per-file key material. |
There was a problem hiding this comment.
key_metadata in table stats files is going to be deprecated here: #17533
Listing where the field is found now might not age well if the list changes by time.
| Encrypted key material is tracked in two places: | ||
|
|
||
| * The `key_metadata` fields in [manifest entries](spec.md#manifests), [manifest list entries](spec.md#manifest-lists), and [statistics files](spec.md#table-metadata-fields) store the per-file key material. | ||
| * The table metadata [`encryption-keys`](spec.md#encryption-keys) list tracks the key hierarchy used to protect that per-file material. |
There was a problem hiding this comment.
If I'm not mistaken encrypted-keys in table metadata is not to protect the key_metadata fields listed above.
|
|
||
| ## Standard Key Metadata | ||
|
|
||
| The `key_metadata` field in manifest entries stores per-file encryption key material as a binary blob. To enable cross-implementation interoperability, the standard encryption scheme defines the following binary format for this field: |
There was a problem hiding this comment.
I think the spec's purpose is cross-implementation interoperability, no need to spell it out: "To enable cross-implementation interoperability". I think we can remove this part of the sentence and just say that the this is the scheme for key_metadata
|
|
||
| ## Standard Key Metadata | ||
|
|
||
| The `key_metadata` field in manifest entries stores per-file encryption key material as a binary blob. To enable cross-implementation interoperability, the standard encryption scheme defines the following binary format for this field: |
There was a problem hiding this comment.
I'd rather remove reference to the table format such as mentioning that manifest entries have a key-metadata field. I think that in the encryption spec we should be more general and not assuming that which table metadata structs have what fields.
|
|
||
| 1. **Key Encryption Keys (KEKs):** Entries where `encrypted-by-id` equals the table's encryption key ID (configured via `encryption.key-id`). The `encrypted-key-metadata` contains the KEK wrapped by the KMS and is opaque to Iceberg — its format is determined by the KMS provider. KEK entries must include a `KEY_TIMESTAMP` property recording the creation time in milliseconds since epoch; this timestamp is used as the AAD when encrypting manifest list key metadata. | ||
|
|
||
| 2. **Manifest List Keys:** Entries where `encrypted-by-id` references a KEK. The `encrypted-key-metadata` contains the Standard Key Metadata (defined above) encrypted with AES GCM using the referenced unwrapped KEK. The ciphertext format is: |
There was a problem hiding this comment.
Saying "Manifest List" is too narrow IMO. In V4, root manifest files, and as a future enhancement, stat files could also store their encryption metadata within this list.
|
|
||
| The usage of the `encryption_key` and `aad_prefix` fields depends on the file format: | ||
|
|
||
| * **AES GCM Stream files**: |
There was a problem hiding this comment.
For a regular reader "AES GCM Stream files" might not mean much in an Iceberg spec. Taking a look at the list below, isn't this AVRO files and Puffin files?
|
|
||
| ### Encryption Key Hierarchy | ||
|
|
||
| The standard encryption scheme uses a two-tier key hierarchy tracked in the table metadata [`encryption-keys`](spec.md#encryption-keys) list: |
There was a problem hiding this comment.
This gives the impression that all encryption related information should go into this structure. I think that what is true instead is that there are 2 different way: 1) raw key-metadata stored per file as described in the above sections, or 2) key-id driven encrypted key-metadata stored together (where sharing between encrypted files is possible).
I think we should articulate this more.
Also, for 2) it's essential to have a KMS. Might make sense to dedicate a sort paragraph to describe what it is and what assumptions we might have.
| @@ -731,7 +664,7 @@ The `data_file` struct consists of the following fields: | |||
| | _optional_ | _optional_ | | ~~**`111 distinct_counts`**~~ | `map<123: int, 124: long>` | **Deprecated. Do not write.** | | |||
| | _optional_ | _optional_ | _optional_ | **`125 lower_bounds`** | `map<126: int, 127: binary>` | Map from column id to lower bound in the column serialized as binary [1]. Each value must be less than or equal to all non-null, non-NaN values in the column for the file [2] | | |||
| | _optional_ | _optional_ | _optional_ | **`128 upper_bounds`** | `map<129: int, 130: binary>` | Map from column id to upper bound in the column serialized as binary [1]. Each value must be greater than or equal to all non-null, non-Nan values in the column for the file [2] | | |||
| | _optional_ | _optional_ | _optional_ | **`131 key_metadata`** | `binary` | Implementation-specific key metadata for encryption | | |||
| | _optional_ | _optional_ | _optional_ | **`131 key_metadata`** | `binary` | Per-file encryption key metadata. See [Standard Key Metadata](encryption-spec.md#standard-key-metadata) for the interoperable format used by the standard encryption scheme. | | |||
There was a problem hiding this comment.
I think this should be enough:
"Per-file encryption key metadata. See Standard Key Metadata."
Same goes for other occurrences.
|
taking a look! i think several people have raise this already. lets see if we can get this moving |
|
I think I'm going to pair this right back to basically just include details of the format for the standard encryption metadata for now and we can always add more detail as we need. |
|
@gaborkaszab @singhpk234 @kevinjqliu @ggershinsky request for comments on this new version please. |
Before this PR
The
key_metadatafields in thedata_fileandmanifest_filestructs are described as "implementation-specific" in the spec. While implementing table encryption in iceberg-rust, we found that the binary format used by the Java implementation is required for cross-implementation interop but isn't documented anywhere - you have to reverse-engineer it from the Java source.See mailing list thread: [DISCUSS] Specifying the encryption key metadata formats for cross-implementation interop
After this PR
Adds a new Encryption spec (
format/encryption-spec.md) documenting Standard Key Metadata, the per-file key metadata format:encryption_key,aad_prefix,file_length)The
key_metadatadescriptions fordata_file(131) andmanifest_file(519) in the table spec now link to it.The key hierarchy (KEKs and manifest list keys stored in
encryption-keys) is intentionally left out and will follow in a separate PR. The statistics filekey-metadatafield is unchanged as it is being deprecated in #17533.