[rustyconover]

Rationale for this change

Closes #1380. Part of apache/arrow#49704 and apache/arrow#54958.

In IPC a dictionary-encoded field is one Field whose type is the value type, so dictionary<values=extension<X>> and extension<X, storage=dictionary<...>> serialize identically. In arrow-go:

• a dictionary of arrow.uuid values is written with the UUID keys on the Field, and reading it back fails (invalid storage type for UUIDType: dictionary);
• writing the schema of an extension type with dictionary storage panics.

The dev@ discussion (https://lists.apache.org/thread/4xj0yrl2vn0lbrv39p3dvd28okvkjxyz) reached lazy consensus on adding custom_metadata: [KeyValue] to DictionaryEncoding for the dictionary value type's metadata.

Companion drafts: Rust apache/arrow-rs#11622, nanoarrow apache/arrow-nanoarrow#972.

This depends on the format change in apache/arrow#54959 (Schema.fbs and the columnar, C Data Interface and integration JSON specs), which is pending a dev@ vote. The C++ implementation and the dictionary_extension archery integration case are in apache/arrow#53994, which is stacked on it. It is the second reference implementation for that vote, so it should not be merged before the format change is accepted.

What changes are included in this PR?

• Generated code: arrow/internal/flatbuf/DictionaryEncoding.go regenerated (flatc 25.12.19) with the new DictionaryEncoding.custom_metadata member.
• IPC writer:
  • The ARROW:extension:name/ARROW:extension:metadata keys of a dictionary's extension value type are written to DictionaryEncoding.custom_metadata instead of Field.custom_metadata.
  • An extension type with dictionary storage keeps writing its keys to Field.custom_metadata, and its schema no longer panics on write.
• IPC reader:
  • Applies DictionaryEncoding.custom_metadata to the value type, wraps the result in the dictionary type, then applies Field.custom_metadata as before.
  • Only registered extension types are applied. Unregistered extension annotations and other unconsumed keys are retained in DictionaryType.ValueMetadata and written back to DictionaryEncoding.custom_metadata, preserving their meaning through a Go read/write round trip.
  • Removing the extension keys from field metadata no longer mutates slices shared with the caller's arrow.Metadata.
• Dictionary types: the new ValueMetadata field keeps value-type metadata separate from field metadata. TypeEqual compares it when CheckMetadata() is requested.
• Integration JSON: an optional "metadata" list in a field's "dictionary" object preserves unknown extension annotations and other value-type metadata. The reader also now honours "isOrdered".
• dictutils.Mapper: finds nested dictionaries inside extension value types, walking through every extension storage layer.
• New arrow/internal/integration package: registers the canonical extension types. It is imported by arrow-json-integration-test, arrow-file-to-stream, arrow-stream-to-file and the Flight integration client and server. The C Data integration library doesn't import it, because its importer does not use the extension registry.

Are these changes tested?

Yes.

• IPC unit tests:
  • schema serialization, checking the keys sit in DictionaryEncoding and not Field;
  • stream and file data round trips of dictionaries of UUID and of an extension type with metadata;
  • extension types with dictionary storage, including a dictionary of extension values as storage;
  • a dictionary nested inside an extension value type, including stream and file round trips where the outer extension or both outer and inner extensions are unregistered;
  • delta dictionaries of dict<uuid>;
  • preservation of unregistered extension annotations and extra non-extension keys, reconstruction after registering the extension, and rejection of registered extensions with the wrong storage type;
  • legacy field-level metadata.
• Integration JSON round trips: dictionary metadata, opaque<dict<uuid>>, list<dict<uuid>>, ordered dictionaries, and unknown extension annotations with extra custom keys through JSON → IPC → JSON.
• Review follow-up validation on EC2 (Linux ARM64, Go 1.26.10): go build -p 8 ./..., go test -p 8 -short -tags assert,test ./..., and race-enabled tests for arrow, arrow/ipc, arrow/internal/arrjson, and arrow/internal/dictutils passed. The full IPC suite also passed with -race -tags assert -count=1 after adding the nested unknown-extension cases.
• Archery, previously run manually on the initial implementation:
  • C++ (GH-49704: [C++] Read and write DictionaryEncoding.custom_metadata for dictionary value types arrow#53994) ⇄ Go passes over IPC and C Data arrays for the extension, dictionary_extension, dictionary, nested_dictionary and extension_union cases.
  • Go's C schema import is skipped there, because the C Data importer does not use the extension registry.
  • Go-to-Go Flight integration passes for the same cases.

Are there any user-facing changes?

Yes:

• Changed IPC output: for a dictionary whose values are an extension type, the extension keys move from Field.custom_metadata to the new DictionaryEncoding.custom_metadata.
  • Such columns now round-trip as dictionary<extension<...>> in Go and in other implementations that support the new member.
  • Readers that predate the format change ignore the member and see a plain dictionary of the storage type. Previously they saw the keys on the field, which typically failed or produced an extension type with dictionary storage.
• Unknown dictionary extensions: readers expose the storage value type while retaining its extension name, serialized metadata, and other custom keys in DictionaryType.ValueMetadata; writing the schema preserves those annotations.
• Integration JSON format: gains an optional "metadata" key in the "dictionary" object.
• Integration programs: the IPC JSON integration, file/stream conversion and Flight integration client/server now register the canonical extension types.

The initial implementation and description were written with Claude Code and reviewed by a human maintainer and an AI reviewer. The metadata-preservation follow-up and additional regression tests were implemented with Codex.

🤖 Generated with Claude Code

[zeroshade]

Thanks — reviewed while still a draft, since it's gated on the apache/arrow#54959 vote. The overall approach looks right to me:

• Writer/reader symmetry is clean: extension keys of a dictionary's value type go to DictionaryEncoding.custom_metadata, and the reader applies them to the value type before wrapping in DictionaryType, so dict<ext> and ext<dict> are no longer conflated. fieldVisitor is per-field, so dictMeta can't leak, and an empty dictMeta writes no member.
• extensionTypeFromMetadata now always builds fresh key/value slices — good catch on the old append(mdkeys[:i], ...) aliasing the caller's metadata.
• Tests pin the behaviour: disabling the reader's use of DictionaryEncoding metadata fails 13 tests in arrow/ipc.

One thing we'd like changed before this lands:

1. DictionaryEncoding.custom_metadata should survive a Go read→write round trip. Today, at field level, an unregistered extension name (and any other key) stays in Field.Metadata, so Go passes it through unchanged. In the new path, anything extensionTypeFromMetadata doesn't consume — unregistered extension names, their serialized metadata, and any non-extension keys — is dropped on read, so reading and re-writing a dict<unregistered-ext> file silently strips the annotation. We'd rather not introduce that pass-through regression, even as a documented limitation. Moving the leftover keys into Field.Metadata wouldn't work, since that reintroduces the dict<ext>/ext<dict> ambiguity on the next write. The leftover metadata needs to be carried with the dictionary's value type so the writer can emit it back into DictionaryEncoding.custom_metadata. How to do that is open — e.g. a field on arrow.DictionaryType for value-type metadata, or an internal placeholder value type that keeps the unregistered name and serialized metadata — happy to discuss whichever you think fits best. A round-trip test with an unregistered extension name plus an extra non-extension key would pin it.
2. Minor: dictutils.Mapper.ImportField unwraps one level of extension value type; an extension whose storage is itself an extension wrapping a nested type wouldn't be descended. Likely unreachable today.

---

This review was drafted by an AI-assisted tool and
confirmed by an Apache Arrow maintainer. After you've
addressed the points above and pushed an update, an Apache Arrow
maintainer — a real person — will take the next look
at the PR. The findings cite the project's review criteria;
if you think one of them is mis-applied, please reply on the
PR and a maintainer will weigh in.

More on how Apache Arrow handles maintainer review:
CONTRIBUTING.md.

[rustyconover]

Hi @zeroshade,

I've read your review and prepared some new code tonight that tests that unrecognized extensions can round trip through.

I've also addressed your concern about nested types.

Rusty