diff --git a/packages/zarr-metadata/README.md b/packages/zarr-metadata/README.md
index a01ed26842..d4f3c6579a 100644
--- a/packages/zarr-metadata/README.md
+++ b/packages/zarr-metadata/README.md
@@ -2,410 +2,46 @@
Documentation:
-Basic tools for modelling Zarr metadata, with minimal dependencies.
+Types, models and validators for Zarr metadata documents, with minimal
+dependencies.
`zarr-metadata` is developed in the
[zarr-python repository](https://github.com/zarr-developers/zarr-python/tree/main/packages/zarr-metadata)
-and released independently of `zarr` itself. Install it with:
+and released independently of `zarr`. Install it with:
```
pip install zarr-metadata
```
-## Who needs this
-
-This library supports software that interacts with Zarr metadata documents.
-
## What this is
-This is *not* a full Zarr implementation. It's a collection of data structures and routines that
-closely model the content of the Zarr specifications, such as:
-
-- **Typed JSON shapes** ([`zarr_metadata.v2`](https://zarr-metadata.readthedocs.io/en/latest/api/v2/) and
- [`zarr_metadata.v3`](https://zarr-metadata.readthedocs.io/en/latest/api/v3/)): `TypedDict` definitions and
- `Literal` aliases for the JSON documents specified by the
- [Zarr v2](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html) and
- [Zarr v3](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html)
- specifications, plus types for
- [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/) and a
- few widely-used-but-unspecified entities (e.g. consolidated metadata).
-- **Document models** ([`zarr_metadata.model`](https://zarr-metadata.readthedocs.io/en/latest/api/model/)): immutable
- models of whole metadata documents, each one a document and the scope
- it was read in, with readers that report every problem at its
- location, and store-key (de)serialization. A document
- produced by `to_json` shares no mutable state with the model that produced
- it.
-- **Definitions and scopes** ([`zarr_metadata.definition`](https://zarr-metadata.readthedocs.io/en/latest/api/definition/),
- with each format's kinds in `zarr_metadata.v3.definition` and
- `zarr_metadata.v2.definition`): each extension's configuration as a
- `TypedDict` with its rules, and the scope that decides which definitions
- read a document. `zarr_metadata.v3` and `zarr_metadata.v2` export every
- definition and scope of their format, such as `BYTES_CODEC` and
- `CORE_AND_EXTENSIONS_CONTEXT`.
-- **JSON type checking** ([`zarr_metadata.typed_json`](https://zarr-metadata.readthedocs.io/en/latest/api/typed_json/)):
- `check` checks a JSON value against any `TypedDict` and reports every
- problem with its location.
-- **Problems** ([`zarr_metadata.problems`](https://zarr-metadata.readthedocs.io/en/latest/api/problems/)): the problem
- values every reader returns, and the error that carries them.
-- **JSON Schema** ([`zarr_metadata.json_schema`](https://zarr-metadata.readthedocs.io/en/latest/api/json_schema/)): the
- JSON Schema of a `TypedDict`, a v3 declaration, or a whole `zarr.json`.
-- **Optional Pydantic integration** ([`zarr_metadata.pydantic`](https://zarr-metadata.readthedocs.io/en/latest/api/pydantic/),
- requires Pydantic 2.13 or newer): each model as a Pydantic field type that
- validates raw documents through the same strict parser.
-
-## What this is for
-
-- **Static Type Checking** The public `TypedDict` definitions describe the static JSON shape of Zarr
- metadata.
-
-- **Dynamic JSON validation** For strict, loc-aware validation of JSON loaded from disk, use the
- model parser:
-
- ```python test="true" session="readme"
- import json
- from zarr_metadata.model import ZarrV3ArrayMetadata
-
- with open("zarr.json", "rb") as f:
- raw = json.load(f)
-
- metadata = ZarrV3ArrayMetadata.from_json(raw)
- ```
-
- The optional Pydantic integration delegates raw input to the same strict
- parser and returns the same normalized model class:
-
- ```python test="true" session="readme"
- from pydantic import TypeAdapter
- import zarr_metadata.pydantic as zmp
-
- adapter: TypeAdapter[zmp.ZarrV3ArrayMetadataField] = TypeAdapter(zmp.ZarrV3ArrayMetadataField)
- metadata = adapter.validate_python(raw)
- encoded = metadata.to_key_value()["zarr.json"]
- ```
-
- A field annotated `Annotated[ZarrV3ArrayMetadata, zmp.InScope(CONTEXT)]`
- is read in `CONTEXT`, whatever the validation context holds, and a model
- instance given to it is read again in `CONTEXT`.
-
- Note that a bare `TypeAdapter` over a public `TypedDict` is a **coercive** shape
- adapter, not a Zarr conformance validator; it may coerce values or discard
- members that the strict model parser rejects. This is better suited for handling input provided
- from Python objects (e.g., user-authored dicts) than JSON.
-
-## Validation boundary
-
-The readers and model constructors enforce the declared document structure and a small set
-of context-free consistency rules, e.g.:
-- fixed format literals,
-- finite JSON numbers outside `attributes` and the members nothing in scope reads
-- non-negative dimensions
-- one `dimension_names` entry per array dimension.
-
-In a v3 document, validation also checks each extension point: the data
-type, the chunk grid, the chunk key encoding, each codec, and each storage
-transformer. Each one is checked by the definition that claims its name in
-a scope. The scope is `CORE_AND_EXTENSIONS_CONTEXT` unless a `context` is passed.
-`ZARR_PYTHON_CONTEXT` adds the data types that zarr-python writes but no
-specification registers.
-If the definition rejects a configuration, the reader rejects it too,
-and a configuration key the definition does not declare is reported as
-`unknown_key`. If no definition in the scope claims a name, the reader
-does not check that extension point, and the consumer decides whether to
-support it.
-
-Some checks relate one part of a v3 document to another:
-
-- The fill value is checked against the data type the document names, by
- that data type's definition.
-- The chunk grid is checked against the shape, by the grid's definition.
-- The codecs are checked as a pipeline. In order, each codec's definition
- checks the codec against the chunk it receives. This includes a shard's
- inner codecs and index codecs.
-
-The readers do no arithmetic on values. For example, they do not check
-whether a fill value survives a `cast_value` round trip.
-
-Some choices that the spec text leaves open, or answers in two ways:
-
-- **`attributes` may hold `NaN`, `Infinity` and `-Infinity`.** The spec
- does not interpret attributes, and zarr-python and xarray write these
- numbers there (for example, a CF `_FillValue`). The models read them,
- and `to_key_value` writes them back as bare tokens, which a strict JSON
- parser rejects. So may a member the spec does not define when no
- definition in scope claims it: nothing reads it, so it is kept as
- written. `check`, from `zarr_metadata.typed_json`, rejects a non-finite
- number anywhere, including in attributes.
-- **A reader walks at most 256 levels of nesting.** A value nested deeper
- is an `invalid_value` problem at the first level past the limit,
- wherever it is. Every reader, writer and comparison uses one stack
- frame per level, and `copy.deepcopy` (and `pickle` before Python 3.12)
- uses two. A document at the limit therefore uses about half of the
- interpreter's default recursion limit, and the rest is left for the
- caller.
-- **`consolidated_metadata: null` is a problem.** A zarr-python 3.0.x bug
- wrote it. The spec requires an object, and the package accepts only an
- object. A reader of those stores removes the key before reading.
-- **An extension name must follow the spec's naming rule**,
- `^[a-z][a-z0-9-_.]+$`, or be a URI, which earlier versions of the spec
- required. Any other name is rejected before any definition is
- consulted, so `""` and `"foo/bar"` are problems, not unknown extensions.
-- **`must_understand: false` is rejected at every extension point**,
- including codecs and storage transformers, although the core spec names
- only the data type, chunk grid and chunk key encoding. A reader that
- skips a codec reads wrong bytes, just as a reader that skips a data type
- reads wrong values. On an unknown top-level member, `must_understand`
- keeps its meaning: a reader can skip that member.
-- **A v2 datetime or timedelta dtype must have a unit**, as in ` Iterator[ValidationProblem]:
+ if "dictionary" in configuration and "dictionary_size" not in configuration:
+ yield ValidationProblem(
+ ("dictionary_size",), "a dictionary needs its size", "missing_key"
+ )
+
+
+ACME_LZ4 = CodecDefinition(
+ name="acme.lz4",
+ configuration=AcmeLz4Configuration,
+ kind="bytes_bytes",
+ size="dynamic",
+ rules=acme_lz4_rules,
+)
+CONTEXT = CORE_AND_EXTENSIONS_CONTEXT.union(ZarrV3Context(codecs=[ACME_LZ4]))
+```
+
+Pass `context=CONTEXT` to any reader or model constructor. A document that
+names `acme.lz4` is then checked against this definition.
+
+## Rules for the pieces
+
+**The `TypedDict`**
+
+- It must be a `typing_extensions.TypedDict`.
+- It must say how undeclared keys are handled: `closed=True`,
+ `extra_items=`, or `closed=False`. An open-by-default `TypedDict` is
+ refused, because it would accept a misspelled key silently.
+
+**Its members** must be JSON shapes:
+
+- `int`, `float`, `bool`, `str`, `None` or `JSONValue`,
+- a `Literal`,
+- `tuple[T, ...]` or `tuple[T1, T2]`,
+- a union,
+- a `TypedDict` or `Mapping[str, V]`,
+- a `NewType` or a type alias.
+
+A number may carry bounds from annotated-types: `Gt`, `Ge`, `Lt`, `Le` or
+`Interval`. Any other `Annotated` metadata is refused when the definition
+is built.
+
+**The rules function**
+
+- It runs only on a configuration within its bounds, so it can rely on
+ them.
+- It reports the location of each problem. The value there becomes the
+ problem's `input`.
+- Show values in messages with `zarr_metadata.problems.shown`. Then they
+ read like the package's own messages.
+
+**Pickling.** Define each function at module top level. A model holds the
+definitions that read it, so the model can be pickled only if they can.
+
+## Other kinds of definition
+
+Other kinds declare more:
+
+- A data type declares its fill value (`fill_value`, `fill_value_rules`,
+ `fill_value_canonical`). It also declares how its values are stored
+ (`storage`).
+- A chunk grid declares which shapes it fits (`shape_rules`) and its chunk
+ lengths (`chunk_lengths`).
+- A codec declares:
+ - what it checks about the chunk it receives (`chunk_rules`),
+ - what it hands on (`transition`),
+ - any pipelines it contains (`pipelines`).
+
+The [`zarr_metadata.v3.definition`](api/v3/definition.md) docstring
+describes each of these.
diff --git a/packages/zarr-metadata/docs/index.md b/packages/zarr-metadata/docs/index.md
index b139dfd307..da545079e3 100644
--- a/packages/zarr-metadata/docs/index.md
+++ b/packages/zarr-metadata/docs/index.md
@@ -1,411 +1,92 @@
# zarr-metadata
-Basic tools for modelling Zarr metadata, with minimal dependencies.
+Types, models and validators for Zarr metadata documents, with minimal
+dependencies.
`zarr-metadata` is developed in the
[zarr-python repository](https://github.com/zarr-developers/zarr-python/tree/main/packages/zarr-metadata)
-and released independently of `zarr` itself. Install it with:
+and released independently of `zarr`. Install it with:
```
pip install zarr-metadata
```
-## Who needs this
-
-This library supports software that interacts with Zarr metadata documents.
-
## What this is
-This is *not* a full Zarr implementation. It's a collection of data structures and routines that
-closely model the content of the Zarr specifications, such as:
-
-- **Typed JSON shapes** ([`zarr_metadata.v2`](api/v2.md) and
- [`zarr_metadata.v3`](api/v3/index.md)): `TypedDict` definitions and
- `Literal` aliases for the JSON documents specified by the
- [Zarr v2](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html) and
- [Zarr v3](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html)
- specifications, plus types for
- [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/) and a
- few widely-used-but-unspecified entities (e.g. consolidated metadata).
-- **Document models** ([`zarr_metadata.model`](api/model.md)): immutable
- models of whole metadata documents, each one a document and the scope
- it was read in, with readers that report every problem at its
- location, and store-key (de)serialization. A document
- produced by `to_json` shares no mutable state with the model that produced
- it.
-- **Definitions and scopes** ([`zarr_metadata.definition`](api/definition.md),
- with each format's kinds in `zarr_metadata.v3.definition` and
- `zarr_metadata.v2.definition`): each extension's configuration as a
- `TypedDict` with its rules, and the scope that decides which definitions
- read a document. `zarr_metadata.v3` and `zarr_metadata.v2` export every
- definition and scope of their format, such as `BYTES_CODEC` and
- `CORE_AND_EXTENSIONS_CONTEXT`.
-- **JSON type checking** ([`zarr_metadata.typed_json`](api/typed_json.md)):
- `check` checks a JSON value against any `TypedDict` and reports every
- problem with its location.
-- **Problems** ([`zarr_metadata.problems`](api/problems.md)): the problem
- values every reader returns, and the error that carries them.
-- **JSON Schema** ([`zarr_metadata.json_schema`](api/json_schema.md)): the
- JSON Schema of a `TypedDict`, a v3 declaration, or a whole `zarr.json`.
-- **Optional Pydantic integration** ([`zarr_metadata.pydantic`](api/pydantic.md),
- requires Pydantic 2.13 or newer): each model as a Pydantic field type that
- validates raw documents through the same strict parser.
-
-## What this is for
-
-- **Static Type Checking** The public `TypedDict` definitions describe the static JSON shape of Zarr
- metadata.
-
-- **Dynamic JSON validation** For strict, location-aware validation of JSON loaded from disk, use the
- model parser:
-
- ```python test="true" session="index"
- import json
- from zarr_metadata.model import ZarrV3ArrayMetadata
-
- with open("zarr.json", "rb") as f:
- raw = json.load(f)
-
- metadata = ZarrV3ArrayMetadata.from_json(raw)
- ```
-
- The optional Pydantic integration delegates raw input to the same strict
- parser and returns the same normalized model class:
-
- ```python test="true" session="index"
- from pydantic import TypeAdapter
- import zarr_metadata.pydantic as zmp
-
- adapter: TypeAdapter[zmp.ZarrV3ArrayMetadataField] = TypeAdapter(zmp.ZarrV3ArrayMetadataField)
- metadata = adapter.validate_python(raw)
- encoded = metadata.to_key_value()["zarr.json"]
- ```
-
- A field annotated `Annotated[ZarrV3ArrayMetadata, zmp.InScope(CONTEXT)]`
- is read in `CONTEXT`, whatever the validation context holds, and a model
- instance given to it is read again in `CONTEXT`.
-
- Note that a bare `TypeAdapter` over a public `TypedDict` is a **coercive** shape
- adapter, not a Zarr conformance validator; it may coerce values or discard
- members that the strict model parser rejects. This is better suited for handling input provided
- from Python objects (e.g., user-authored dicts) than JSON.
-
-## Validation boundary
-
-The readers and model constructors enforce the declared document structure and a small set
-of context-free consistency rules, e.g.:
-- fixed format literals,
-- finite JSON numbers outside `attributes` and the members nothing in scope reads
-- non-negative dimensions
-- one `dimension_names` entry per array dimension.
-
-In a v3 document, validation also checks each extension point: the data
-type, the chunk grid, the chunk key encoding, each codec, and each storage
-transformer. Each one is checked by the definition that claims its name in
-a scope. The scope is `CORE_AND_EXTENSIONS_CONTEXT` unless a `context` is passed.
-`ZARR_PYTHON_CONTEXT` adds the data types that zarr-python writes but no
-specification registers.
-If the definition rejects a configuration, the reader rejects it too,
-and a configuration key the definition does not declare is reported as
-`unknown_key`. If no definition in the scope claims a name, the reader
-does not check that extension point, and the consumer decides whether to
-support it.
-
-Making the context and explicit part of metadata parsing allows different libraries, which might support different
-sets of Zarr features, to formally represent exactly the codecs, data types, etc they support when
-parsing metadata.
-
-Some checks relate one part of a v3 document to another:
-
-- The fill value is checked against the data type the document names, by
- that data type's definition.
-- The chunk grid is checked against the shape, by the grid's definition.
-- The codecs are checked as a pipeline. In order, each codec's definition
- checks the codec against the chunk it receives. This includes a shard's
- inner codecs and index codecs.
-
-The readers do no arithmetic on values. For example, they do not check
-whether a fill value survives a `cast_value` round trip.
-
-Some choices that the spec text leaves open, or answers in two ways:
+A collection of data structures and routines that closely model the
+metadata rules in the [Zarr v2](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html)
+and [Zarr v3](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html)
+specifications and in [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/):
-- **`attributes` may hold `NaN`, `Infinity` and `-Infinity`.** The spec
- does not interpret attributes, and zarr-python and xarray write these
- numbers there (for example, a CF `_FillValue`). The models read them,
- and `to_key_value` writes them back as bare tokens, which a strict JSON
- parser rejects. So may a member the spec does not define when no
- definition in scope claims it: nothing reads it, so it is kept as
- written. `check`, from `zarr_metadata.typed_json`, rejects a non-finite
- number anywhere, including in attributes.
-- **A reader walks at most 256 levels of nesting.** A value nested deeper
- is an `invalid_value` problem at the first level past the limit,
- wherever it is. Every reader, writer and comparison uses one stack
- frame per level, and `copy.deepcopy` (and `pickle` before Python 3.12)
- uses two. A document at the limit therefore uses about half of the
- interpreter's default recursion limit, and the rest is left for the
- caller.
-- **`consolidated_metadata: null` is a problem.** A zarr-python 3.0.x bug
- wrote it. The spec requires an object, and the package accepts only an
- object. A reader of those stores removes the key before reading.
-- **An extension name must follow the spec's naming rule**,
- `^[a-z][a-z0-9-_.]+$`, or be a URI, which earlier versions of the spec
- required. Any other name is rejected before any definition is
- consulted, so `""` and `"foo/bar"` are problems, not unknown extensions.
-- **`must_understand: false` is rejected at every extension point**,
- including codecs and storage transformers, although the core spec names
- only the data type, chunk grid and chunk key encoding. A reader that
- skips a codec reads wrong bytes, just as a reader that skips a data type
- reads wrong values. On an unknown top-level member, `must_understand`
- keeps its meaning: a reader can skip that member.
-- **A v2 datetime or timedelta dtype must have a unit**, as in ` configuration["length_bytes"]:
+ yield ValidationProblem((), "expected fewer bytes than the type holds", "invalid_value")
+
+
+PACKED = DataTypeDefinition(
+ name="acme.packed",
+ configuration=PackedConfiguration,
+ fill_value=Base64Bytes,
+ fill_value_rules=packed_fill_value_rules,
+)
+context = CORE_AND_EXTENSIONS_CONTEXT.union(ZarrV3Context(data_types=[PACKED]))
+data_type, _ = resolve(
+ {"name": "acme.packed", "configuration": {"length_bytes": 2}}, DataTypeDefinition, context
+)
+for fill_value in ("YWI=", "not base64!", "YWJj"):
+ print(fill_value, [problem.message for problem in fill_value_problems(data_type, fill_value)])
+# YWI= []
+# not base64! ['expected standard-alphabet base64, got "not base64!"']
+# YWJj ['expected fewer bytes than the type holds']
+```
+
+A reader of a document in this context checks its fill value with these rules, and so does a pydantic field. A JSON Schema written for the context does not contain the rules, so it accepts any string here.
-[^1]: The one exception is the `r*` data type, which is identified by a templated string with no `configuration` object. This mixes the identifier with the configuration in the same JSON value, which is not good design. So this library normalizes `r*` into the same form as all the other metadata objects: the declaration `"r16"` keeps its name, and is read by the `r*` definition with the configuration `{"bits": 16}`.
+[^1]: The one exception is the [`r*` data type](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#data-types), which is identified by a templated string with no `configuration` object. This mixes the identifier with the configuration in the same JSON value, which is not good design. So this library normalizes `r*` into the same form as all the other metadata objects. The declaration `"r16"` keeps its name. The `r*` definition reads it with the configuration `{"bits": 16}`.
diff --git a/packages/zarr-metadata/docs/pydantic.md b/packages/zarr-metadata/docs/pydantic.md
new file mode 100644
index 0000000000..5316c00534
--- /dev/null
+++ b/packages/zarr-metadata/docs/pydantic.md
@@ -0,0 +1,66 @@
+# Pydantic
+
+`zarr_metadata.pydantic` is an optional integration.
+
+- It requires Pydantic 2.13 or newer.
+- The core package does not depend on Pydantic.
+
+Each exported name is a Pydantic field type over one of the models. A raw
+document given to the field goes through the same strict reader as
+`from_json`. The value of the field is an instance of the model class
+itself.
+
+```python test="true" session="pydantic"
+import json
+from pydantic import BaseModel, TypeAdapter
+import zarr_metadata.pydantic as zmp
+
+with open("zarr.json", "rb") as f:
+ raw = json.load(f)
+
+adapter: TypeAdapter[zmp.ZarrV3ArrayMetadataField] = TypeAdapter(zmp.ZarrV3ArrayMetadataField)
+metadata = adapter.validate_python(raw)
+encoded = metadata.to_key_value()["zarr.json"]
+
+
+class ArrayManifest(BaseModel):
+ path: str
+ metadata: zmp.ZarrV3ArrayMetadataField
+
+
+manifest = ArrayManifest.model_validate({"path": "a/b", "metadata": raw})
+```
+
+A static type checker sees `manifest.metadata` as a
+`zarr_metadata.model.ZarrV3ArrayMetadata`.
+
+A failed read raises a Pydantic `ValidationError`. It has one line error
+per problem, in the format Pydantic uses for its own errors.
+
+## Choosing the context
+
+A field type reads its document in the context given by Pydantic's
+validation context:
+
+- If the validation context is a `Context`, every field type of that
+ format reads in it.
+- If it is a mapping:
+ - the v3 field types read in its `"zarr_metadata_context"` item;
+ - the v2 field types read in its `"zarr_metadata_context_v2"` item.
+- If no context is given for a format, its field types read in that
+ format's default: `CORE_AND_EXTENSIONS_CONTEXT` or `CORE_V2_CONTEXT`.
+
+To pin a field to one context, annotate it with `InScope`. A field
+annotated `Annotated[ZarrV3ArrayMetadata, zmp.InScope(CONTEXT)]` is read in
+`CONTEXT`, whatever the validation context holds. A model instance given to
+it is read again in `CONTEXT`.
+
+## A bare `TypeAdapter` is not a validator
+
+A `TypeAdapter` over one of the public `TypedDict`s is a **coercive** shape
+adapter. It is not a Zarr conformance validator. It may coerce values, or
+discard fields that the strict reader rejects.
+
+- It suits input built from Python objects, such as user-authored dicts.
+- For JSON read from a store, use the field types above, or the readers
+ directly.
diff --git a/packages/zarr-metadata/docs/repair.md b/packages/zarr-metadata/docs/repair.md
new file mode 100644
index 0000000000..f4f197e40e
--- /dev/null
+++ b/packages/zarr-metadata/docs/repair.md
@@ -0,0 +1,59 @@
+# Repairing known writer bugs
+
+The readers are strict. If a known writer bug made a document invalid, the
+readers report problems.
+
+The repair readers in `zarr_metadata.model` handle these documents:
+
+- `read_repaired_node_metadata_v3`
+- `read_repaired_node_metadata_v2`
+- `read_repaired_consolidated_metadata_v2`
+
+Each one works in three steps:
+
+1. Undo the known bugs.
+2. Read the result strictly.
+3. Return each change it made as a `Repair`.
+
+## Bugs the repair readers undo
+
+- **A chunk length of 0 along an empty dimension**, in a v3
+ [`regular` chunk grid](https://zarr-specs.readthedocs.io/en/latest/v3/chunk-grids/regular-grid/index.html#description).
+ zarr-python 3.0 and 3.1 wrote it. The repair writes 1.
+- **`consolidated_metadata: null` on a v3 group.** zarr-python 3.0 and 3.1
+ wrote it. The spec
+ [requires an object](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L802-L816).
+ The repair removes the key.
+- **A `consolidated_metadata` member in a `.zgroup` entry of a
+ `.zmetadata`**, below the root. zarr-python 3 writes it. The
+ [v2 spec](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html#groups)
+ allows no key but `zarr_format` in a `.zgroup`. The repair removes the
+ member.
+- **A bare `NaN`, `Infinity` or `-Infinity` as a v2 fill value.**
+ zarr-python 3.0.0 to 3.0.6 wrote it. The repair writes the string the
+ [v2 spec](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html#fill-value-encoding)
+ uses, such as `"NaN"`.
+- **A bare non-finite float or complex fill value in a v3 group's
+ consolidated metadata.** zarr-python 3.0.0 to 3.0.6 wrote it. The repair
+ writes the string the
+ [v3 spec](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/data-types/index.html#permitted-fill-values)
+ uses, such as `"NaN"`.
+
+## Bugs the repair readers do not undo
+
+A document with one of these bugs is refused:
+
+- **The fill value of an `S` or `V` array as a JSON array of byte values**,
+ in a `.zmetadata` entry. zarr-python 3.0.7 to 3.0.10 wrote it. The
+ array's own `.zarray` file holds the
+ [base64 string](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html#fill-value-encoding)
+ the v2 spec requires, and the package reads that file.
+- **The fill value of a v2 datetime array as an ISO 8601 string**, not as a
+ count of ticks. zarr-python 3.0.7 and 3.0.8 wrote it.
+- **A chunk length of 0 along a dimension that is not empty**, in a v3
+ `regular` chunk grid. zarr-python 3.0 and 3.1 wrote it for an explicit
+ chunk length of 0.
+- **A chunk length of `true` or `false`** in a v3 `regular` chunk grid.
+ Some releases of zarr-python from 3.0 to 3.2 wrote `true` for
+ `chunks=True`. Some releases of 3.0 and 3.1 wrote `false` for
+ `chunks=False`.
diff --git a/packages/zarr-metadata/docs/validation-rules.md b/packages/zarr-metadata/docs/validation-rules.md
new file mode 100644
index 0000000000..5a0b194c21
--- /dev/null
+++ b/packages/zarr-metadata/docs/validation-rules.md
@@ -0,0 +1,148 @@
+# What the readers check
+
+The readers and model constructors enforce the declared document
+structure. They also check a small set of consistency rules that need no
+context. For example:
+
+- fixed format literals
+ ([`zarr_format`](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#zarr-format)
+ and
+ [`node_type`](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#node-type)),
+- finite JSON numbers, except in `attributes` and in members that nothing
+ in the context reads,
+- non-negative
+ [dimensions](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#array-metadata-shape),
+- one
+ [`dimension_names`](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#dimension-names)
+ entry per array dimension.
+
+## Extension points
+
+In a v3 document, the readers also check each
+[extension point](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#extension-points):
+
+- the data type,
+- the chunk grid,
+- the chunk key encoding,
+- each codec,
+- each storage transformer.
+
+The definition that claims a name in the context checks that extension
+point. The context is `CORE_AND_EXTENSIONS_CONTEXT` unless you pass a
+`context`.
+
+- If the definition rejects a configuration, the reader rejects it too.
+- A configuration key the definition does not declare is reported as
+ `unknown_key`.
+- If no definition claims a name, the reader does not check that extension
+ point. The consumer decides whether to support it.
+
+The [Context](context.md) page shows this with an example.
+
+## Checks across the document
+
+Some checks relate one part of a v3 document to another:
+
+- The
+ [fill value](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#fill-value)
+ is checked against the data type the document names. The data type's
+ definition does this check.
+- The
+ [chunk grid](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#chunk-grids)
+ is checked against the shape. The grid's definition does this check.
+- The codecs are checked as a
+ [pipeline](https://github.com/zarr-developers/zarr-specs/blob/v3.1.0/docs/v3/core/index.rst#L931-L1024).
+ In order, each codec's definition checks the codec against the chunk it
+ receives. This includes a
+ [shard's](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/codecs/sharding-indexed/index.html#configuration-parameters)
+ inner codecs and index codecs.
+
+The readers do no arithmetic on values. For example, they do not check
+whether a fill value survives a `cast_value` round trip.
+
+## Decisions the spec leaves open
+
+The spec leaves some choices open, or answers them in two ways. Here is
+what the package does.
+
+**`attributes` may hold `NaN`, `Infinity` and `-Infinity`.**
+
+- The spec
+ [does not interpret attributes](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#attributes).
+- zarr-python and xarray write these numbers there, for example as a CF
+ `_FillValue`.
+- The models read them. `to_key_value` writes them back as bare tokens,
+ which a strict JSON parser rejects.
+- The same goes for a member the spec does not define, when no definition
+ in the context claims it. Nothing reads it, so it is kept as written.
+- `check`, from `zarr_metadata.typed_json`, rejects a non-finite number
+ anywhere, including in attributes.
+
+**A reader walks at most 256 levels of nesting.**
+
+- A value nested deeper is an `invalid_value` problem at the first level
+ past the limit, wherever it is.
+- Every reader, writer and comparison uses one stack frame per level.
+ `copy.deepcopy` uses two, and so does `pickle` before Python 3.12.
+- So a document at the limit uses about half of the interpreter's default
+ recursion limit. The rest is left for the caller.
+
+**`consolidated_metadata: null` is a problem.**
+
+- A zarr-python 3.0.x bug wrote it.
+- The spec
+ [requires an object](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L802-L816),
+ and the package accepts only an object.
+- The [repair readers](repair.md) remove the key before reading.
+
+**An extension name must follow the spec's
+[naming rule](https://github.com/zarr-developers/zarr-specs/blob/v3.1.0/docs/v3/core/index.rst#L1571-L1607)**,
+`^[a-z][a-z0-9-_.]+$`, or be a URI.
+
+- Earlier versions of the spec required URIs.
+- Any other name is rejected before any definition is consulted.
+- So `""` and `"foo/bar"` are problems, not unknown extensions.
+
+**`must_understand: false` is rejected at every extension point**,
+including codecs and storage transformers.
+
+- The
+ [core spec](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#must-understand)
+ names only the data type, chunk grid and chunk key encoding.
+- But a reader that skips a codec reads wrong bytes. That is just as bad as
+ a reader that skips a data type and reads wrong values.
+- On an unknown top-level member, `must_understand` keeps its meaning: a
+ reader can skip that member.
+
+**A v3 `regular` chunk grid's chunk lengths must be at least 1**, even
+along a dimension of length 0.
+
+- The
+ [regular grid spec](https://zarr-specs.readthedocs.io/en/latest/v3/chunk-grids/regular-grid/index.html#description)
+ says "Chunk sizes must be greater than zero".
+ [zarr-specs#379](https://github.com/zarr-developers/zarr-specs/pull/379)
+ added that sentence after v3.1.0.
+- The
+ [core spec's wording](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#concepts-and-terminology)
+ requires less: "non-zero when the corresponding dimensions of the arrays
+ have non-zero length". But it does not explicitly allow anything more.
+- So the package rejects a chunk length of 0. zarr-python 3.0 and 3.1 wrote
+ one for an empty dimension.
+
+**A v2 datetime or timedelta dtype must have a unit**, as in ` zarr_metadata.definition': api/definition.md
diff --git a/packages/zarr-metadata/src/zarr_metadata/_json.py b/packages/zarr-metadata/src/zarr_metadata/_json.py
index 69829d32f4..d5de1537a2 100644
--- a/packages/zarr-metadata/src/zarr_metadata/_json.py
+++ b/packages/zarr-metadata/src/zarr_metadata/_json.py
@@ -20,7 +20,14 @@
from zarr_metadata._common import JSONValue
from zarr_metadata._sentinel import UNSET
-ProblemKind = Literal["missing_key", "invalid_type", "invalid_value", "invalid_json", "unknown_key"]
+ProblemKind = Literal[
+ "missing_key",
+ "invalid_type",
+ "invalid_value",
+ "invalid_json",
+ "unknown_key",
+ "unclaimed_declaration",
+]
"""Machine-readable classification of a `ValidationProblem`.
- `missing_key`: a required key (document key or store key) is absent.
@@ -35,6 +42,10 @@
open since 2023), and many readers reject such a key. It has its own
kind so that a caller who tolerates it can distinguish it from a wrong
value, and so that it never hides other problems in the same object.
+- `unclaimed_declaration`: a declaration that nothing in scope claims, which
+ a reader must understand to open the node. Only
+ `read_openable_node_metadata_v3` reports it: the other readers accept an
+ unclaimed declaration, since that is what keeps the format open.
"""
diff --git a/packages/zarr-metadata/src/zarr_metadata/definition/__init__.py b/packages/zarr-metadata/src/zarr_metadata/definition/__init__.py
index 019725679a..2ba4055ba6 100644
--- a/packages/zarr-metadata/src/zarr_metadata/definition/__init__.py
+++ b/packages/zarr-metadata/src/zarr_metadata/definition/__init__.py
@@ -116,40 +116,42 @@
a closure cannot be pickled, and a `functools.partial` can be pickled
but does not compare equal to its loaded copy.
- from collections.abc import Iterator
- from typing import Annotated, NotRequired
+```python
+from collections.abc import Iterator
+from typing import Annotated, NotRequired
- from annotated_types import Ge
- from typing_extensions import TypedDict
+from annotated_types import Ge
+from typing_extensions import TypedDict
- from zarr_metadata import ValidationProblem
- from zarr_metadata.definition import Nested
- from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition, ZarrV3Context
+from zarr_metadata import ValidationProblem
+from zarr_metadata.definition import Nested
+from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition, ZarrV3Context
- class AcmeLz4Configuration(TypedDict, closed=True):
- acceleration: Annotated[int, Ge(1)]
- dictionary: NotRequired[str]
- dictionary_size: NotRequired[Annotated[int, Ge(1)]]
+class AcmeLz4Configuration(TypedDict, closed=True):
+ acceleration: Annotated[int, Ge(1)]
+ dictionary: NotRequired[str]
+ dictionary_size: NotRequired[Annotated[int, Ge(1)]]
- def acme_lz4_rules(
- configuration: AcmeLz4Configuration, nested: Nested
- ) -> Iterator[ValidationProblem]:
- if "dictionary" in configuration and "dictionary_size" not in configuration:
- yield ValidationProblem(
- ("dictionary_size",), "a dictionary needs its size", "missing_key"
- )
+def acme_lz4_rules(
+ configuration: AcmeLz4Configuration, nested: Nested
+) -> Iterator[ValidationProblem]:
+ if "dictionary" in configuration and "dictionary_size" not in configuration:
+ yield ValidationProblem(
+ ("dictionary_size",), "a dictionary needs its size", "missing_key"
+ )
- ACME_LZ4 = CodecDefinition(
- name="acme.lz4",
- configuration=AcmeLz4Configuration,
- kind="bytes_bytes",
- size="dynamic",
- rules=acme_lz4_rules,
- )
- CONTEXT = CORE_AND_EXTENSIONS_CONTEXT.union(ZarrV3Context(codecs=[ACME_LZ4]))
+ACME_LZ4 = CodecDefinition(
+ name="acme.lz4",
+ configuration=AcmeLz4Configuration,
+ kind="bytes_bytes",
+ size="dynamic",
+ rules=acme_lz4_rules,
+)
+CONTEXT = CORE_AND_EXTENSIONS_CONTEXT.union(ZarrV3Context(codecs=[ACME_LZ4]))
+```
The TypedDict declares how keys it does not declare are handled. With
`closed=True`, such a key is a problem, as above. With `extra_items=`,
diff --git a/packages/zarr-metadata/src/zarr_metadata/json_schema.py b/packages/zarr-metadata/src/zarr_metadata/json_schema.py
index 2296b78652..7ff4bde29d 100644
--- a/packages/zarr-metadata/src/zarr_metadata/json_schema.py
+++ b/packages/zarr-metadata/src/zarr_metadata/json_schema.py
@@ -15,13 +15,15 @@
accepts a number with no fractional part, such as `1.0`, as an integer,
while `check` requires `1`:
- from zarr_metadata.json_schema import json_schema
-
- json_schema(GzipCodecConfiguration)
- # {'$schema': 'https://json-schema.org/draft/2020-12/schema',
- # 'type': 'object',
- # 'properties': {'level': {'type': 'integer', 'minimum': 0, 'maximum': 9}},
- # 'required': ['level'], 'additionalProperties': False}
+```python
+from zarr_metadata.json_schema import json_schema
+
+json_schema(GzipCodecConfiguration)
+# {'$schema': 'https://json-schema.org/draft/2020-12/schema',
+# 'type': 'object',
+# 'properties': {'level': {'type': 'integer', 'minimum': 0, 'maximum': 9}},
+# 'required': ['level'], 'additionalProperties': False}
+```
**One v3 declaration.** `declaration_json_schema_v3(kind, context=None)` writes one declaration
read as `kind` in the scope `context`: a declaration that a definition in
diff --git a/packages/zarr-metadata/src/zarr_metadata/model/__init__.py b/packages/zarr-metadata/src/zarr_metadata/model/__init__.py
index 0228861744..2501347273 100644
--- a/packages/zarr-metadata/src/zarr_metadata/model/__init__.py
+++ b/packages/zarr-metadata/src/zarr_metadata/model/__init__.py
@@ -25,7 +25,12 @@
`read_repaired_node_metadata_v3` or `read_repaired_node_metadata_v2`.
They undo each bug with `repair_node_metadata_v3` or
`repair_node_metadata_v2` before the strict read, and report what they
-changed. `zarr_metadata.json_schema` writes a JSON Schema of what the v3
+changed. A reader accepts a declaration that nothing in its scope claims
+(an unknown codec, or an unknown top-level member), and a model lists each
+one a reader must understand in `unclaimed_declarations`. To also report
+each one as a problem, so that a reading has no problems exactly when a
+reader can open the node, use `read_openable_node_metadata_v3`.
+`zarr_metadata.json_schema` writes a JSON Schema of what the v3
readers read, except for the rules. One declaration on its own is read with
`zarr_metadata.definition.resolve`.
@@ -91,6 +96,7 @@
read_consolidated_metadata_v3,
read_group_metadata_v3,
read_node_metadata_v3,
+ read_openable_node_metadata_v3,
)
from zarr_metadata.model._repair import (
Repair,
@@ -160,6 +166,7 @@
"read_group_metadata_v3",
"read_node_metadata_v2",
"read_node_metadata_v3",
+ "read_openable_node_metadata_v3",
"read_repaired_consolidated_metadata_v2",
"read_repaired_node_metadata_v2",
"read_repaired_node_metadata_v3",
diff --git a/packages/zarr-metadata/src/zarr_metadata/model/_array_v3.py b/packages/zarr-metadata/src/zarr_metadata/model/_array_v3.py
index 061819fb3d..fa175ed005 100644
--- a/packages/zarr-metadata/src/zarr_metadata/model/_array_v3.py
+++ b/packages/zarr-metadata/src/zarr_metadata/model/_array_v3.py
@@ -443,14 +443,14 @@ def must_understand_subset(
}
-def waived_members(values: Mapping[str, ZarrV3ExtensionField]) -> list[str]:
+def waived_members(values: Mapping[str, object]) -> list[str]:
"""The keys of the extra fields in `values` (by key, as written) that say `"must_understand": false`, which a reader may ignore."""
return [key for key, value in values.items() if _waived(value)]
-def _waived(value: ZarrV3ExtensionField) -> bool:
+def _waived(value: object) -> bool:
"""Whether `value`, an extra field as written, is an object that says `"must_understand": false`."""
- return isinstance(value, Mapping) and value.get("must_understand") is False
+ return is_object(value) and value.get("must_understand") is False
class ZarrV3ArrayMetadataUpdate(TypedDict, total=False, closed=True):
@@ -835,6 +835,10 @@ def from_json(
Raises `MetadataValidationError` with every problem the read finds.
`read_array_metadata_v3` returns the reading this model is built
from, and the problems of a document that has any.
+ A declaration that nothing in `context` claims, such as an unknown
+ codec or an unknown top-level member, is not a problem; the model
+ lists each one a reader must understand in `unclaimed_declarations`.
+ `read_openable_node_metadata_v3` reports each one as a problem.
"""
return cls(data, context=context)
@@ -919,7 +923,9 @@ def read_array_metadata_v3(
type. Unknown top-level keys are allowed (they map to
`extra_fields`). A reader must understand each one that does not
declare `must_understand: false`; the model reports these as
- `must_understand_fields`.
+ `must_understand_fields`, and every declaration a reader must
+ understand, at any depth, as `unclaimed_declarations`.
+ `read_openable_node_metadata_v3` reports each of those as a problem.
To apply a policy over the declarations (for example, allow only the core
spec's declarations), iterate over the reading's `declarations()`. A value that
diff --git a/packages/zarr-metadata/src/zarr_metadata/model/_group_v3.py b/packages/zarr-metadata/src/zarr_metadata/model/_group_v3.py
index 42f3e61357..82aad927ab 100644
--- a/packages/zarr-metadata/src/zarr_metadata/model/_group_v3.py
+++ b/packages/zarr-metadata/src/zarr_metadata/model/_group_v3.py
@@ -450,7 +450,10 @@ def from_json(
scope that claims the member, as `CORE_CONTEXT` does, this is a problem,
because the spec requires an object. Use
`read_repaired_node_metadata_v3` to read such a store. A member the
- spec does not define is kept in `extra_fields`.
+ spec does not define is kept in `extra_fields`, and is not a
+ problem when nothing in `context` claims it; the model lists each
+ such member a reader must understand in `unclaimed_declarations`.
+ `read_openable_node_metadata_v3` reports each one as a problem.
"""
return cls(data, context=context)
@@ -769,6 +772,44 @@ def read_node_metadata_v3(
return ZarrV3UnknownNodeMetadataReading(problems)
+def read_openable_node_metadata_v3(
+ value: object, *, context: ZarrV3Context | None = None
+) -> ZarrV3NodeMetadataReading:
+ """Read `value`, a v3 `zarr.json`, in `context` as `read_node_metadata_v3` does, and also report each declaration a reader must understand to open the node that nothing in `context` claims.
+
+ The spec requires a reader to refuse a node with a declaration it does
+ not understand: an unknown codec, data type or chunk grid, or an
+ unknown top-level member that does not say `"must_understand": false`.
+ `read_node_metadata_v3` accepts these, as `UnclaimedDeclaration`s,
+ since that is what keeps the format open. This reader reports each one
+ as a problem of kind `unclaimed_declaration`, at its location, and
+ gives no model when there is one. A member that says
+ `"must_understand": false` is ignored whole, with every declaration in
+ it, as `unclaimed_declarations` ignores it. So the reading has no
+ problems exactly when a reader that understands `context` can open the
+ node. The documents a group's `consolidated_metadata` lists are not
+ read, as `read_group_metadata_v3` does not read them.
+ """
+ reading = read_node_metadata_v3(value, context=context)
+ # A node reading is only of an array or a group when `value` is an object.
+ if isinstance(reading, ZarrV3UnknownNodeMetadataReading) or not is_object(value):
+ return reading
+ waived = waived_members({key: value[key] for key in reading.extra_fields})
+ unclaimed = [
+ ValidationProblem(
+ loc,
+ f"nothing in scope claims the {kind_name(declaration.read_as)} {declaration.name!r}, "
+ "which a reader must understand to open the node",
+ "unclaimed_declaration",
+ )
+ for loc, declaration in unclaimed_declarations_of(reading.declarations(), waived).items()
+ ]
+ if len(unclaimed) == 0:
+ return reading
+ problems = (*reading.problems, *with_input(unclaimed, value))
+ return dataclasses.replace(reading, problems=problems, metadata=None)
+
+
ZarrV3NodeMetadata = TypeAliasType("ZarrV3NodeMetadata", ZarrV3ArrayMetadata | ZarrV3GroupMetadata)
"""The model of a v3 `zarr.json`: an array model or a group model, as its `node_type` declares."""
@@ -875,7 +916,8 @@ def read_group_metadata_v3(
Unknown top-level keys are allowed (they map to `extra_fields`). A
reader must understand each one that does not declare
`must_understand: false`; the model reports these as
- `must_understand_fields`. A `consolidated_metadata` member, if
+ `must_understand_fields`, and `read_openable_node_metadata_v3` reports
+ each one as a problem. A `consolidated_metadata` member, if
present and claimed by the scope, as `CORE_CONTEXT` claims it, is checked
against the envelope the core spec fixes. The documents it lists are
not read; `read_consolidated_metadata_v3` reads them. This is the
diff --git a/packages/zarr-metadata/src/zarr_metadata/pydantic.py b/packages/zarr-metadata/src/zarr_metadata/pydantic.py
index 4434660d8f..308519c0ef 100644
--- a/packages/zarr-metadata/src/zarr_metadata/pydantic.py
+++ b/packages/zarr-metadata/src/zarr_metadata/pydantic.py
@@ -29,9 +29,11 @@
For example:
- TypeAdapter(zmp.ZarrV3ArrayMetadataField).validate_python(document, context=CONTEXT)
- ArrayManifest.model_validate(data, context={"zarr_metadata_context": CONTEXT})
- Mixed.model_validate(data, context={"zarr_metadata_context": V3, "zarr_metadata_context_v2": V2})
+```python
+TypeAdapter(zmp.ZarrV3ArrayMetadataField).validate_python(document, context=CONTEXT)
+ArrayManifest.model_validate(data, context={"zarr_metadata_context": CONTEXT})
+Mixed.model_validate(data, context={"zarr_metadata_context": V3, "zarr_metadata_context_v2": V2})
+```
An existing model instance passes through unchanged when no scope is
given. When one is, the instance is read again in it with `refined_in`, so
@@ -63,11 +65,13 @@
Usage:
- import zarr_metadata.pydantic as zmp
+```python
+import zarr_metadata.pydantic as zmp
- class ArrayManifest(BaseModel):
- path: str
- metadata: zmp.ZarrV3ArrayMetadataField
+class ArrayManifest(BaseModel):
+ path: str
+ metadata: zmp.ZarrV3ArrayMetadataField
+```
Static type checkers see each field type as its core model class, so
`manifest.metadata` is a `zarr_metadata.model.ZarrV3ArrayMetadata`.
diff --git a/packages/zarr-metadata/src/zarr_metadata/typed_json.py b/packages/zarr-metadata/src/zarr_metadata/typed_json.py
index 30ed95f695..982a3d4868 100644
--- a/packages/zarr-metadata/src/zarr_metadata/typed_json.py
+++ b/packages/zarr-metadata/src/zarr_metadata/typed_json.py
@@ -6,14 +6,16 @@
value against the TypedDict and returns a value of the TypedDict (or
None) and every problem, each located in the value:
- import json
+```python
+import json
- from zarr_metadata import ZarrV3ArrayMetadataJSON
- from zarr_metadata.typed_json import check
+from zarr_metadata import ZarrV3ArrayMetadataJSON
+from zarr_metadata.typed_json import check
- document, problems = check(json.loads(raw), ZarrV3ArrayMetadataJSON)
- for problem in problems:
- print(problem.loc, problem.kind, problem.message)
+document, problems = check(json.loads(raw), ZarrV3ArrayMetadataJSON)
+for problem in problems:
+ print(problem.loc, problem.kind, problem.message)
+```
`check` reads a TypedDict as the typing spec defines it, regardless of
how its module writes annotations:
diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/__init__.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/__init__.py
index 528215df03..a1da0f902a 100644
--- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/__init__.py
+++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/__init__.py
@@ -37,6 +37,7 @@
BytesDataTypeName,
BytesFillValue,
base64_bytes,
+ base64_fill_value_rules,
)
from zarr_metadata.v3.data_type.complex64 import (
COMPLEX64_DATA_TYPE,
@@ -338,6 +339,7 @@
"ZarrPythonRawBytesDataTypeName",
"ZarrPythonRawBytesFillValue",
"base64_bytes",
+ "base64_fill_value_rules",
"hex_float16",
"hex_float32",
"hex_float64",
diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/_byte.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/_byte.py
index cfc7adddec..f9eda83e16 100644
--- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/_byte.py
+++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/_byte.py
@@ -28,6 +28,23 @@ def base64_problems(value: str) -> Iterator[ValidationProblem]:
)
+def base64_fill_value_rules(
+ configuration: object, nested: object, value: object
+) -> Iterator[ValidationProblem]:
+ """Check that a fill value written as a string is standard-alphabet base64: a data type's `fill_value_rules`.
+
+ `Base64Bytes` is a `NewType` of `str`, so a fill value annotated with
+ it accepts any string. Pass this rule as the data type's
+ `fill_value_rules` to reject a string that is not base64. A value of
+ any other shape is not checked here, so the rule also serves a fill
+ value that may be written another way, such as
+ `tuple[ByteValue, ...] | Base64Bytes`. A rule that checks more, such
+ as the number of bytes, yields from this one first.
+ """
+ if isinstance(value, str):
+ yield from base64_problems(value)
+
+
def base64_decoded(value: str) -> bytes | None:
"""The bytes that `value` encodes in standard-alphabet base64; None when it is not such base64."""
try:
@@ -41,4 +58,10 @@ def base64_spelling(data: bytes) -> str:
return base64.b64encode(data).decode("ascii")
-__all__ = ["ByteValue", "base64_decoded", "base64_problems", "base64_spelling"]
+__all__ = [
+ "ByteValue",
+ "base64_decoded",
+ "base64_fill_value_rules",
+ "base64_problems",
+ "base64_spelling",
+]
diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/bytes.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/bytes.py
index b92facd4af..f7e5c572aa 100644
--- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/bytes.py
+++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/bytes.py
@@ -5,10 +5,8 @@
"""
import base64
-from collections.abc import Iterator
from typing import Final, Literal
-from zarr_metadata._json import ValidationProblem
from zarr_metadata._numpy import Base64Bytes, base64_bytes
from zarr_metadata.definition._definition import (
EmptyConfiguration,
@@ -18,7 +16,11 @@
DataTypeDefinition,
variable_length,
)
-from zarr_metadata.v3.data_type._byte import ByteValue, base64_problems, base64_spelling
+from zarr_metadata.v3.data_type._byte import (
+ ByteValue,
+ base64_fill_value_rules,
+ base64_spelling,
+)
BYTES_DATA_TYPE_NAME: Final = "bytes"
"""The `data_type` value for the variable-length `bytes` type."""
@@ -34,14 +36,6 @@
"""
-def _fill_value_rules(
- configuration: EmptyConfiguration, nested: Nested, value: BytesFillValue
-) -> Iterator[ValidationProblem]:
- """Check that a string fill value is standard-alphabet base64. A fill value written as byte values is not checked here."""
- if isinstance(value, str):
- yield from base64_problems(value)
-
-
def _fill_value_canonical(
configuration: EmptyConfiguration, nested: Nested, value: BytesFillValue
) -> str:
@@ -60,7 +54,7 @@ def _fill_value_canonical(
name=BYTES_DATA_TYPE_NAME,
configuration=EmptyConfiguration,
fill_value=BytesFillValue,
- fill_value_rules=_fill_value_rules,
+ fill_value_rules=base64_fill_value_rules,
fill_value_canonical=_fill_value_canonical,
storage=variable_length,
)
@@ -79,4 +73,5 @@ def _fill_value_canonical(
"BytesDataTypeName",
"BytesFillValue",
"base64_bytes",
+ "base64_fill_value_rules",
]
diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/variable_length_bytes.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/variable_length_bytes.py
index 28803e9d80..11c7da2e69 100644
--- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/variable_length_bytes.py
+++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/variable_length_bytes.py
@@ -10,14 +10,12 @@
"""
import base64
-from collections.abc import Iterator
from typing import Final, Literal
-from zarr_metadata._json import ValidationProblem
from zarr_metadata._numpy import Base64Bytes
from zarr_metadata.definition._definition import EmptyConfiguration, Nested
from zarr_metadata.v3._definition import DataTypeDefinition, variable_length
-from zarr_metadata.v3.data_type._byte import base64_problems, base64_spelling
+from zarr_metadata.v3.data_type._byte import base64_fill_value_rules, base64_spelling
VARIABLE_LENGTH_BYTES_DATA_TYPE_NAME: Final = "variable_length_bytes"
"""The `data_type` value for the `variable_length_bytes` type."""
@@ -29,13 +27,6 @@
"""Permitted JSON shape of the `fill_value` field for `variable_length_bytes`: a `Base64Bytes` string, which is what zarr-python writes and reads."""
-def _fill_value_rules(
- configuration: EmptyConfiguration, nested: Nested, value: VariableLengthBytesFillValue
-) -> Iterator[ValidationProblem]:
- """Check that the fill value is standard-alphabet base64."""
- yield from base64_problems(value)
-
-
def _fill_value_canonical(
configuration: EmptyConfiguration, nested: Nested, value: VariableLengthBytesFillValue
) -> str:
@@ -47,7 +38,7 @@ def _fill_value_canonical(
name=VARIABLE_LENGTH_BYTES_DATA_TYPE_NAME,
configuration=EmptyConfiguration,
fill_value=VariableLengthBytesFillValue,
- fill_value_rules=_fill_value_rules,
+ fill_value_rules=base64_fill_value_rules,
fill_value_canonical=_fill_value_canonical,
storage=variable_length,
)
diff --git a/packages/zarr-metadata/tests/model/test_unclaimed_declarations.py b/packages/zarr-metadata/tests/model/test_unclaimed_declarations.py
index 02b5521882..e66914b6d2 100644
--- a/packages/zarr-metadata/tests/model/test_unclaimed_declarations.py
+++ b/packages/zarr-metadata/tests/model/test_unclaimed_declarations.py
@@ -7,6 +7,7 @@
import pytest
from typing_extensions import TypedDict
+from zarr_metadata import UNSET
from zarr_metadata.definition import UnclaimedDeclaration
from zarr_metadata.model import (
ZarrV2ArrayMetadata,
@@ -15,6 +16,8 @@
ZarrV3ArrayMetadata,
ZarrV3GroupMetadata,
ZarrV3NodeMetadata,
+ read_node_metadata_v3,
+ read_openable_node_metadata_v3,
)
from zarr_metadata.v3 import CORE_CONTEXT
from zarr_metadata.v3.definition import (
@@ -190,3 +193,78 @@ def test_node_type_tells_a_type_checker_which_model_a_node_is() -> None:
for node in nodes:
if node.zarr_format == 3 and node.node_type == "group":
assert node.consolidated() is None
+
+
+@pytest.mark.parametrize(
+ ("document", "context", "unclaimed", "problems"),
+ [
+ (ARRAY, CORE_CONTEXT, set(), set()),
+ ({**ARRAY, "codecs": ("bytes", ZSTD)}, CORE_CONTEXT, {("codecs", 1)}, set()),
+ (
+ {**ARRAY, "codecs": (SHARD,)},
+ CORE_CONTEXT,
+ {("codecs", 0, "configuration", "codecs", 1)},
+ set(),
+ ),
+ ({**ARRAY, "acme": 1}, CORE_CONTEXT, {("acme",)}, set()),
+ ({**ARRAY, "acme": {"must_understand": False}}, CORE_CONTEXT, set(), set()),
+ ({**GROUP, "acme": {"name": "acme"}}, CORE_CONTEXT, {("acme",)}, set()),
+ ({**GROUP, "acme": {"must_understand": False}}, CORE_CONTEXT, set(), set()),
+ (
+ {**ARRAY, "acme.mirror": {"codec": {"name": "acme.zip"}}},
+ MIRRORING,
+ {("acme.mirror", "codec")},
+ set(),
+ ),
+ (
+ {**ARRAY, "acme.mirror": {"codec": {"name": "acme.zip"}, "must_understand": False}},
+ MIRRORING,
+ set(),
+ set(),
+ ),
+ (
+ {**ARRAY, "fill_value": "x", "codecs": ("bytes", ZSTD)},
+ CORE_CONTEXT,
+ {("codecs", 1)},
+ {("fill_value",)},
+ ),
+ ({**GROUP, "node_type": "acme"}, CORE_CONTEXT, set(), {("node_type",)}),
+ ("not a node", CORE_CONTEXT, set(), {()}),
+ ],
+ ids=[
+ "all-claimed",
+ "a-codec",
+ "a-codec-inside-a-shard",
+ "an-extra-member",
+ "a-waived-extra-member",
+ "a-group-member",
+ "a-waived-group-member",
+ "a-codec-inside-a-claimed-member",
+ "a-codec-inside-a-claimed-member-that-waives",
+ "with-other-problems",
+ "an-unknown-node-type",
+ "not-an-object",
+ ],
+)
+def test_read_openable_node_metadata_v3_reports_what_a_reader_must_understand(
+ document: object,
+ context: Any, # noqa: ANN401 - a v3 scope
+ unclaimed: set[tuple[str | int, ...]],
+ problems: set[tuple[str | int, ...]],
+) -> None:
+ """`read_openable_node_metadata_v3` keeps every problem `read_node_metadata_v3` finds and adds one of kind `unclaimed_declaration` at the location of each declaration `unclaimed_declarations` lists, with what the document writes there as its input, whether or not the document has other problems. It gives a model exactly when there are no problems, and the model is the one `read_node_metadata_v3` gives."""
+ plain = read_node_metadata_v3(document, context=context)
+ reading = read_openable_node_metadata_v3(document, context=context)
+ assert reading.problems[: len(plain.problems)] == plain.problems
+ assert {problem.loc for problem in plain.problems} == problems
+ added = reading.problems[len(plain.problems) :]
+ assert {problem.loc for problem in added} == unclaimed
+ assert all(problem.kind == "unclaimed_declaration" for problem in added)
+ assert all(problem.input is not UNSET for problem in added)
+ if plain.metadata is not None:
+ assert set(plain.metadata.unclaimed_declarations) == unclaimed
+ if len(reading.problems) == 0:
+ assert reading.metadata == plain.metadata
+ assert reading.metadata is not None
+ else:
+ assert reading.metadata is None
diff --git a/packages/zarr-metadata/tests/test_docs.py b/packages/zarr-metadata/tests/test_docs.py
index 7726151126..a05596075d 100644
--- a/packages/zarr-metadata/tests/test_docs.py
+++ b/packages/zarr-metadata/tests/test_docs.py
@@ -8,12 +8,14 @@
from __future__ import annotations
+import textwrap
from collections import defaultdict
from pathlib import Path
import pytest
from pytest_examples import CodeExample, EvalExample, find_examples
+import zarr_metadata.definition
from zarr_metadata.model import ZarrV3ArrayMetadata
PACKAGE_ROOT = Path(__file__).parent.parent
@@ -71,3 +73,31 @@ def test_every_python_block_says_whether_it_runs() -> None:
)
]
assert unmarked == []
+
+
+def test_the_extensions_page_example_is_the_definition_module_docstring_example() -> None:
+ """The acme.lz4 example on the extensions page is, line for line, the example in the `zarr_metadata.definition` module docstring. Only the page's copy runs under these tests, so this is what keeps the docstring's copy from drifting."""
+ page_blocks = [
+ example
+ for example in _examples()
+ if example.path == PACKAGE_ROOT / "docs" / "extensions.md"
+ and example.prefix_settings().get("session") == "extensions"
+ ]
+ assert len(page_blocks) == 1, (
+ "expected exactly one extensions-session block on docs/extensions.md"
+ )
+ docstring = zarr_metadata.definition.__doc__
+ assert docstring is not None, "zarr_metadata.definition has no module docstring"
+ lines = docstring.splitlines()
+ marker = "from collections.abc import Iterator"
+ starts = [i for i, line in enumerate(lines) if line.strip() == marker]
+ assert len(starts) == 1, f"expected one line {marker!r} in the docstring, found {len(starts)}"
+ assert lines[starts[0] - 1].strip() == "```python", (
+ f"expected {marker!r} to open a ```python fence in the docstring"
+ )
+ end = starts[0]
+ while end < len(lines) and lines[end].strip() != "```":
+ end += 1
+ assert end < len(lines), "the docstring's ```python fence is not closed"
+ docstring_example = textwrap.dedent("\n".join(lines[starts[0] : end])).strip()
+ assert page_blocks[0].source.strip().splitlines() == docstring_example.splitlines()
diff --git a/packages/zarr-metadata/tests/v3/data_type/bytes/test_validators.py b/packages/zarr-metadata/tests/v3/data_type/bytes/test_validators.py
index 7f41d878f8..c104b81bf4 100644
--- a/packages/zarr-metadata/tests/v3/data_type/bytes/test_validators.py
+++ b/packages/zarr-metadata/tests/v3/data_type/bytes/test_validators.py
@@ -1,16 +1,16 @@
-"""Cover the `base64_bytes` brand validator.
+"""Cover the `base64_bytes` brand validator and the `base64_fill_value_rules` fill value rule.
The pydantic-driven fixture tests don't enforce the base64 alphabet or
length-multiple-of-4 constraint because `Base64Bytes` is a `NewType`,
which pydantic treats as plain `str`. Direct coverage of the validator
-function lives here.
+function and the rule built on it lives here.
"""
from __future__ import annotations
import pytest
-from zarr_metadata.v3.data_type.bytes import base64_bytes
+from zarr_metadata.v3.data_type.bytes import base64_bytes, base64_fill_value_rules
VALID = [
"", # empty is valid base64 (length 0, multiple of 4)
@@ -41,3 +41,16 @@ def test_valid(value: str) -> None:
def test_invalid(value: str) -> None:
with pytest.raises(ValueError, match="standard-alphabet base64"):
base64_bytes(value)
+
+
+# A value written in another shape, such as byte values, is the annotation's to check.
+@pytest.mark.parametrize("value", [*VALID, (1, 2, 3), None])
+def test_base64_fill_value_rules_accepts(value: object) -> None:
+ assert list(base64_fill_value_rules({}, (), value)) == []
+
+
+@pytest.mark.parametrize("value", INVALID)
+def test_error_base64_fill_value_rules_rejects_a_string_that_is_not_base64(value: str) -> None:
+ assert [(p.loc, p.kind) for p in base64_fill_value_rules({}, (), value)] == [
+ ((), "invalid_value")
+ ]