Repository navigation
Add custom_metadata to IPC DictionaryEncoding - #11622
Open
rustyconover wants to merge 1 commit into
Open
rustyconover wants to merge 1 commit into
rustyconover wants to merge 1 commit into
Conversation
2 of 5 tasks
github-actions
Bot
added
arrow-ipc
This was referenced Oct 16, 2026
rustyconover
marked this pull request as ready for review
October 21, 2026 15:45
alamb
added
the
Prolific Author
rustyconover
force-pushed
the
gh-49704-dictionary-value-metadata
branch
from
October 26, 2026 15:52
3ab2539 to
afa15b8
Compare
Adds the draft `DictionaryEncoding.custom_metadata` member from apache/arrow#49704, which holds metadata for the dictionary value type (for example the extension type of a dictionary whose values are an extension type), to `format/Schema.fbs` and the generated code (flatc 25.2.10). `DataType::Dictionary` cannot carry metadata for its value type (apache#7982), so until it can the IPC readers copy the value type metadata into the field's metadata, without replacing keys already present, unless the field has its own `ARROW:extension:name`, in which case it is dropped. The integration JSON reader maps the dictionary's optional `metadata` attribute, and the C Data Interface import maps the metadata of an imported schema's `dictionary`, the same way, so all three readers agree. The IPC writer and the C Data export are unchanged: dictionary field metadata is still written to the field itself. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
rustyconover
force-pushed
the
gh-49704-dictionary-value-metadata
branch
from
October 28, 2026 16:49
afa15b8 to
0ba33b7
Compare
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
Rationale for this change
In IPC today, "dictionary of an extension type" and "extension type with dictionary storage" serialize identically, because
Field.custom_metadatais the only place metadata can go. The format change in apache/arrow#54959 (see apache/arrow#49704) addsDictionaryEncoding.custom_metadatafor the dictionary value type, in format version 1.6. arrow-rs needs to be able to read it.Merging this should wait for the format change to be accepted in apache/arrow. Implementations for other languages: C++ apache/arrow#53994, Go apache/arrow-go#1381, nanoarrow apache/arrow-nanoarrow#972.
What changes are included in this PR?
format/Schema.fbsand the generatedarrow-ipc/src/gen/Schema.rsgain the new member (generated code matches flatc 25.2.10).DataType::Dictionarycan't hold value type metadata yet (Field metadata for dictionary-encoded extension types is lost #7982). Until it can, that metadata is copied onto theField, unless the field already has its ownARROW:extension:name. Existing field keys win. This keeps today's behavior, where e.g.arrow.uuidshows up on dictionary fields.Are these changes tested?
Yes. Unit tests cover the mapping, key precedence, nesting in list/struct, empty vectors, invalid UTF-8, and reading via
StreamReaderandFileReader. Integration JSON and C Data import tests cover the same cases. I also checked with archery that arrow-rs reads the C++ and Godictionary_extensionoutput.Are there any significant user-facing changes?
Fieldmetadata. C Data import previously dropped it; malformed metadata there is now an error.arrow_ipc::DictionaryEncodingArgshas a new public field,custom_metadata. Struct literals need to add it or use..Default::default().Written with help from Claude Code and reviewed by me.