feat(zarr-metadata): export base64_fill_value_rules for data type definitions

packages/zarr-metadata/changes/4604.feature.md

@@ -0,0 +1 @@
+`zarr_metadata.model.read_openable_node_metadata_v3` reads a v3 `zarr.json` as `read_node_metadata_v3` does, and also reports each declaration a reader must understand to open the node that nothing in scope claims, such as an unknown codec or an unknown top-level member without `"must_understand": false`, as a problem of the new kind `unclaimed_declaration`. Its reading has no problems exactly when a reader that understands the scope can open the node, so a validator does not have to check `unclaimed_declarations` itself. The other readers still accept unclaimed declarations, and the `from_json` and reader docstrings now point to `unclaimed_declarations` and the new reader.

packages/zarr-metadata/changes/4605.doc.md

@@ -0,0 +1 @@
+The documentation now has a short landing page that points each kind of reader to a guide page, instead of one long page. New guide pages cover models and readers, the rules the readers check, consolidated metadata, repairing known writer bugs, writing an extension, JSON Schema and Pydantic. Each claim the pages make about a Zarr specification links to the section of the spec that makes it, and the consolidated metadata page says what the specs do and do not settle, with links to the zarr-specs pull requests and issues and to the original zarr-python implementation. The pages say "context" for a `Context` where some said "scope".

packages/zarr-metadata/changes/4609.feature.md

@@ -0,0 +1 @@
+Export `base64_fill_value_rules` from `zarr_metadata.v3.data_type.bytes`, the rule that rejects a fill value string that is not standard-alphabet base64, so a data type defined outside the package can pass it as `fill_value_rules` for a `Base64Bytes` fill value, or yield from it in a rule of its own. The guide to parsing and validation now shows how to define a data type with `fill_value_rules`.

packages/zarr-metadata/docs/api/index.md

@@ -9,7 +9,7 @@ The package is organized to mirror the structure of the Zarr specifications:
 - [`zarr_metadata.json_schema`](json_schema.md) — JSON Schema output for
   any `TypedDict`, one v3 declaration, or a whole v3 `zarr.json`
 - [`zarr_metadata.model`](model.md) — document models (each one a
-  document and the scope it was read in), the readers that return
+  document and the context it was read in), the readers that return
   problems instead of raising them, and the `UNSET` sentinel
 - [`zarr_metadata.problems`](problems.md) — `ValidationProblem`,
   `ProblemKind`, `Problems` and `MetadataValidationError`, which every
@@ -21,7 +21,7 @@ The package is organized to mirror the structure of the Zarr specifications:
   typing spec defines them, with every problem located
 - [`zarr_metadata.v2`](v2.md) — `TypedDict` shapes for Zarr v2 documents
   (`.zarray`, `.zgroup`, `.zattrs`, `.zmetadata`), and
-  `zarr_metadata.v2.definition`: the v2 kinds of metadata, the scope
+  `zarr_metadata.v2.definition`: the v2 kinds of metadata, the context
   `CORE_V2_CONTEXT`, and the readers of one `dtype` or codec. Each dtype family's
   definition is in `zarr_metadata.v2.data_type`, and each codec's in
   `zarr_metadata.v2.codec`
@@ -32,10 +32,11 @@ The package is organized to mirror the structure of the Zarr specifications:
 - [`zarr_metadata.definition`](definition.md) — what both formats share
   for reading a declaration: `Definition`, `resolve`, the three results of a
   reading (`AcceptedDeclaration`, `UnclaimedDeclaration`, `RefusedDeclaration`), and the
-  scope type `Context`. Its module docstring is the guide
+  context type `Context`. Its module docstring is the guide
 - [`zarr_metadata.v3.definition`](v3/definition.md) — the five v3 kinds
   of metadata (`CodecDefinition`, `DataTypeDefinition`, ...), the codec
-  pipeline types, and the v3 scopes `CORE_CONTEXT` and `CORE_AND_EXTENSIONS_CONTEXT`
+  pipeline types, and the v3 contexts `CORE_CONTEXT`,
+  `CORE_AND_EXTENSIONS_CONTEXT` and `ZARR_PYTHON_CONTEXT`
 
 The document types, models, and spec vocabulary — including the store keys —
 are re-exported at the top level, so

packages/zarr-metadata/docs/consolidated-metadata.md

@@ -0,0 +1,143 @@
+# Consolidated metadata
+
+Consolidated metadata is a copy of every metadata document below a group,
+kept in one place. A reader can then open a whole hierarchy with one read.
+
+Consolidated metadata is a convention of zarr-python, the reference
+implementation. Neither Zarr specification defines it in full. This page
+first says what is settled, then describes the API.
+
+## Status in the specifications
+
+**Zarr v2.** The v2 spec has no consolidated metadata.
+
+- The `.zmetadata` file was designed in zarr-python
+  [PR 268](https://github.com/zarr-developers/zarr-python/pull/268). It
+  merged in November 2018 and shipped in zarr-python 2.3.0.
+- The
+  [zarr-python 2 tutorial](https://zarr.readthedocs.io/en/v2.18.3/tutorial.html#consolidating-metadata)
+  describes it.
+- zarr-specs [issue 113](https://github.com/zarr-developers/zarr-specs/issues/113),
+  on standardizing the format, is open.
+
+**Zarr v3.** The core spec names the member but does not define its
+entries.
+
+- zarr-specs [PR 309](https://github.com/zarr-developers/zarr-specs/pull/309)
+  proposes the `consolidated_metadata` member, with the `kind: "inline"`
+  envelope that zarr-python 3 writes. It follows
+  [issue 136](https://github.com/zarr-developers/zarr-specs/issues/136). It
+  has been open since August 2024 and is not merged.
+- [PR 373](https://github.com/zarr-developers/zarr-specs/pull/373) merged
+  in July 2026. Since then, the core spec names the member "for historical
+  reasons" and fixes its envelope
+  ([spec text](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L802-L816)).
+  The envelope is an object with:
+    - `must_understand: false`,
+    - `kind: "inline"`,
+    - a `metadata` object.
+- PR 373 closed
+  [issue 371](https://github.com/zarr-developers/zarr-specs/issues/371).
+  That issue found a problem with the v3.1.0 extension rules. Read
+  strictly, they put every consolidated group zarr-python 3 had written out
+  of spec, because the envelope has no `name`.
+
+This package reads the envelope the spec fixes. It reads the entries as
+zarr-python writes them.
+
+## Zarr v3
+
+Consolidated metadata holds the hierarchy below its group, with the group
+as the root.
+
+- The document of the node at `/a/b` is stored under the key `a/b`. The
+  [stored representation](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#stored-representation)
+  strips the leading `/` of a path.
+- The documents and the group must form a tree. Only groups have child
+  nodes, and the parent of every node is present.
+  `ZarrV3ConsolidatedMetadata` checks this when it reads the documents.
+
+`NodeName` and `NodePath`, in `zarr_metadata.v3`, are string types. They
+satisfy the spec's rules for
+[node names](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#node-names)
+and paths. They are modeled on the zarrs types with the same names. These
+functions check a string against those rules:
+
+- `validate_node_name_v3`, `is_node_name_v3` and `parse_node_name_v3`,
+- the matching `node_path` functions.
+
+### Reading
+
+A group does not read the documents its `consolidated_metadata` lists.
+Consolidated metadata is a convention of the reference implementation, not
+core metadata. So the group checks the
+[envelope that the core spec fixes](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L802-L816),
+then holds the member as written. `CORE_CONTEXT` holds
+`CONSOLIDATED_METADATA_EXTRA_FIELD`, the definition that reads this
+envelope.
+
+A reader can use the member as a source of child documents, in place of
+the store. There are two ways:
+
+- Read one document with `read_node_metadata_v3` when that node is opened.
+- Read all of them at once, with one of:
+    - the group's `consolidated()`,
+    - `ZarrV3ConsolidatedMetadata`,
+    - `read_consolidated_metadata_v3`, which returns problems instead of
+      raising them.
+
+Either way, a document with a problem fails only when it is read.
+
+Which context the documents are read in:
+
+- `consolidated()` uses the group's own context.
+- The other three take a `context`. Pass `context=group.context` to use the
+  group's own context. With no `context`, they use
+  `CORE_AND_EXTENSIONS_CONTEXT`.
+
+### Writing
+
+To write consolidated metadata:
+
+1. Build a `ZarrV3ConsolidatedMetadata`.
+2. Pass it to the group's `update`.
+
+The group reads the listed models in its own context. It refuses a model
+that its context reads differently. Then it writes the document.
+
+`ZarrV3ConsolidatedMetadata` reads every document strictly. It also
+accepts node models as entries. It accepts such an entry when the entry's
+claims refine into the model's context. Otherwise it refuses the entry at
+its path. To consolidate children read in several contexts, consolidate
+them in the `union` of those contexts.
+
+## Zarr v2
+
+A v2 hierarchy's consolidated metadata is the `.zmetadata` document at the
+root of the store. zarr-python 2.3 and later write it. It is an object
+with:
+
+- `zarr_consolidated_format: 1`,
+- a `metadata` object. This maps each metadata file's path to that file's
+  contents, under keys such as `a/.zarray`, `a/.zattrs` and `a/b/.zgroup`.
+
+The format comes from zarr-python
+[PR 268](https://github.com/zarr-developers/zarr-python/pull/268). No
+specification defines it.
+
+`ZarrV2ConsolidatedMetadata` reads such a document.
+`read_consolidated_metadata_v2` does the same, but returns problems instead
+of raising them. Both take a `context`. The default is `CORE_V2_CONTEXT`.
+
+How the entries are read:
+
+- Each `.zarray` or `.zgroup` entry is read as a node, merged with its
+  sibling `.zattrs`.
+- A `.zattrs` with no sibling is kept, but forms no node.
+- Any other entry must be JSON, and is kept.
+
+`to_json` returns the entries keyed by file, as written.
+
+The `.zmetadata` that zarr-python 3 writes for a v2 hierarchy has a known
+bug. `read_repaired_consolidated_metadata_v2` undoes it. See
+[Repairing known writer bugs](repair.md).

packages/zarr-metadata/docs/context.md

@@ -4,7 +4,7 @@ Resolving metadata documents requires a context, a `Context`, that defines which
 
 ## Why do we need a context
 
-The Zarr V3 spec defines the JSON structure of `zarr.json` for arrays and groups exactly, by defining the fields that must be present and the JSON types they must have. But the __interpretation__ of those fields as actual data types, codecs, chunk grids, etc is not strictly defined by the spec. Relating a JSON value like `"bytes"` to the `bytes` codec requires a mapping from names to codec types.
+The Zarr V3 spec defines the JSON structure of `zarr.json` for arrays and groups exactly. It defines the members that must be present and the JSON types they must have. But the __interpretation__ of those members as actual data types, codecs, chunk grids, etc is not strictly defined by the spec. Relating a JSON value like `"bytes"` to the `bytes` codec requires a mapping from names to codec types.
 
 `zarr-metadata` makes this explicit by validating JSON documents with a __context object__. That context object contains definitions of codecs, data types, etc, which can be matched to the declarations in the JSON document.
 
@@ -47,7 +47,7 @@ document: ZarrV3ArrayMetadataJSON = {
 }
 ```
 
-In the above code snippet we define a `zarr.json` document with three codec declarations (we will just call them codecs for brevity): one named `bytes`, another named `gzip`, and a third named `unknown`. The `bytes` codec here is correctly configured according to the `bytes` codec spec. The `gzip` declaration here violates the `gzip` codec spec because the `level` configuration field is too high. And the `unknown` codec is unknown, it has no specification and cannot be resolved.
+In the above code snippet we define a `zarr.json` document with three codec declarations. (We will just call them codecs for brevity.) They are named `bytes`, `gzip` and `unknown`. The `bytes` codec here is correctly configured according to the `bytes` codec spec. The `gzip` declaration here violates the `gzip` codec spec because the `level` configuration member is too high. And the `unknown` codec is unknown, it has no specification and cannot be resolved.
 
 These three codecs represent three different scenarios that this library handles, two with distinct failure modes. We demonstrate how this works in the next code snippet.
 
@@ -101,7 +101,7 @@ These are the three scenarios:
 
 - `bytes` is parsed to an `AcceptedDeclaration`. A definition in the context claims the name `bytes`, and the configuration passes its checks.
 - The malformed `gzip` is parsed to a `RefusedDeclaration`. A definition claims the name `gzip`, but the configuration fails one of its checks: a `level` of 99 is outside the range gzip allows. The read reports the problem, with its location in the document, and builds no model, so `reading.metadata` is `None`. Building the model directly raises the same problem as a `MetadataValidationError`.
-- The `unknown` codec is parsed to an `UnclaimedDeclaration`. Nothing in the context claims the name, so it is kept as written and not checked. This is not a problem: the spec lets a document name codecs a reader does not know.
+- The `unknown` codec is parsed to an `UnclaimedDeclaration`. Nothing in the context claims the name, so it is kept as written and not checked. This is not a problem: the spec [lets a document name codecs](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#extension-codecs) defined outside it, which a reader may not know.
 
 So one JSON document can be valid in one context and invalid in another. This is why a model is a document _and_ the context it was read in: the context defines the interpretation of the document. A model keeps its context, and moving it to another context reads the document again. Our `small` model was valid in its original context, but it is not valid in the new one:
 

packages/zarr-metadata/docs/extensions.md

@@ -0,0 +1,116 @@
+# Writing an extension
+
+A v3 document may name a codec, data type, chunk grid, chunk key encoding
+or storage transformer that this package does not define. Each of these is
+an
+[extension point](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#extension-points)
+of the v3 spec.
+
+The readers leave such a declaration unclaimed. To check it, a program
+adds a **definition** to the context it reads in.
+
+The [`zarr_metadata.definition`](api/definition.md) module docstring is the
+full reference. This page is the short version.
+
+A definition is a value, not a class to subclass. It has:
+
+- a `name`,
+- a `TypedDict` for the configuration's JSON,
+- a `rules` function, when the type cannot express every rule. The function
+  yields the spec violations in a configuration.
+
+A v3 codec definition also says:
+
+- what kind of codec it is,
+- whether its output size is fixed by its input size.
+
+```python test="true" session="extensions"
+from collections.abc import Iterator
+from typing import Annotated, NotRequired
+
+from annotated_types import Ge
+from typing_extensions import TypedDict
+
+from zarr_metadata import ValidationProblem
+from zarr_metadata.definition import Nested
+from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition, ZarrV3Context
+
+
+class AcmeLz4Configuration(TypedDict, closed=True):
+    acceleration: Annotated[int, Ge(1)]
+    dictionary: NotRequired[str]
+    dictionary_size: NotRequired[Annotated[int, Ge(1)]]
+
+
+def acme_lz4_rules(
+    configuration: AcmeLz4Configuration, nested: Nested
+) -> Iterator[ValidationProblem]:
+    if "dictionary" in configuration and "dictionary_size" not in configuration:
+        yield ValidationProblem(
+            ("dictionary_size",), "a dictionary needs its size", "missing_key"
+        )
+
+
+ACME_LZ4 = CodecDefinition(
+    name="acme.lz4",
+    configuration=AcmeLz4Configuration,
+    kind="bytes_bytes",
+    size="dynamic",
+    rules=acme_lz4_rules,
+)
+CONTEXT = CORE_AND_EXTENSIONS_CONTEXT.union(ZarrV3Context(codecs=[ACME_LZ4]))
+```
+
+Pass `context=CONTEXT` to any reader or model constructor. A document that
+names `acme.lz4` is then checked against this definition.
+
+## Rules for the pieces
+
+**The `TypedDict`**
+
+- It must be a `typing_extensions.TypedDict`.
+- It must say how undeclared keys are handled: `closed=True`,
+  `extra_items=`, or `closed=False`. An open-by-default `TypedDict` is
+  refused, because it would accept a misspelled key silently.
+
+**Its members** must be JSON shapes:
+
+- `int`, `float`, `bool`, `str`, `None` or `JSONValue`,
+- a `Literal`,
+- `tuple[T, ...]` or `tuple[T1, T2]`,
+- a union,
+- a `TypedDict` or `Mapping[str, V]`,
+- a `NewType` or a type alias.
+
+A number may carry bounds from annotated-types: `Gt`, `Ge`, `Lt`, `Le` or
+`Interval`. Any other `Annotated` metadata is refused when the definition
+is built.
+
+**The rules function**
+
+- It runs only on a configuration within its bounds, so it can rely on
+  them.
+- It reports the location of each problem. The value there becomes the
+  problem's `input`.
+- Show values in messages with `zarr_metadata.problems.shown`. Then they
+  read like the package's own messages.
+
+**Pickling.** Define each function at module top level. A model holds the
+definitions that read it, so the model can be pickled only if they can.
+
+## Other kinds of definition
+
+Other kinds declare more:
+
+- A data type declares its fill value (`fill_value`, `fill_value_rules`,
+  `fill_value_canonical`). It also declares how its values are stored
+  (`storage`).
+- A chunk grid declares which shapes it fits (`shape_rules`) and its chunk
+  lengths (`chunk_lengths`).
+- A codec declares:
+    - what it checks about the chunk it receives (`chunk_rules`),
+    - what it hands on (`transition`),
+    - any pipelines it contains (`pipelines`).
+
+The [`zarr_metadata.v3.definition`](api/v3/definition.md) docstring
+describes each of these.