diff --git a/packages/zarr-metadata/README.md b/packages/zarr-metadata/README.md index a01ed26842..d4f3c6579a 100644 --- a/packages/zarr-metadata/README.md +++ b/packages/zarr-metadata/README.md @@ -2,410 +2,46 @@ Documentation: -Basic tools for modelling Zarr metadata, with minimal dependencies. +Types, models and validators for Zarr metadata documents, with minimal +dependencies. `zarr-metadata` is developed in the [zarr-python repository](https://github.com/zarr-developers/zarr-python/tree/main/packages/zarr-metadata) -and released independently of `zarr` itself. Install it with: +and released independently of `zarr`. Install it with: ``` pip install zarr-metadata ``` -## Who needs this - -This library supports software that interacts with Zarr metadata documents. - ## What this is -This is *not* a full Zarr implementation. It's a collection of data structures and routines that -closely model the content of the Zarr specifications, such as: - -- **Typed JSON shapes** ([`zarr_metadata.v2`](https://zarr-metadata.readthedocs.io/en/latest/api/v2/) and - [`zarr_metadata.v3`](https://zarr-metadata.readthedocs.io/en/latest/api/v3/)): `TypedDict` definitions and - `Literal` aliases for the JSON documents specified by the - [Zarr v2](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html) and - [Zarr v3](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html) - specifications, plus types for - [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/) and a - few widely-used-but-unspecified entities (e.g. consolidated metadata). -- **Document models** ([`zarr_metadata.model`](https://zarr-metadata.readthedocs.io/en/latest/api/model/)): immutable - models of whole metadata documents, each one a document and the scope - it was read in, with readers that report every problem at its - location, and store-key (de)serialization. A document - produced by `to_json` shares no mutable state with the model that produced - it. -- **Definitions and scopes** ([`zarr_metadata.definition`](https://zarr-metadata.readthedocs.io/en/latest/api/definition/), - with each format's kinds in `zarr_metadata.v3.definition` and - `zarr_metadata.v2.definition`): each extension's configuration as a - `TypedDict` with its rules, and the scope that decides which definitions - read a document. `zarr_metadata.v3` and `zarr_metadata.v2` export every - definition and scope of their format, such as `BYTES_CODEC` and - `CORE_AND_EXTENSIONS_CONTEXT`. -- **JSON type checking** ([`zarr_metadata.typed_json`](https://zarr-metadata.readthedocs.io/en/latest/api/typed_json/)): - `check` checks a JSON value against any `TypedDict` and reports every - problem with its location. -- **Problems** ([`zarr_metadata.problems`](https://zarr-metadata.readthedocs.io/en/latest/api/problems/)): the problem - values every reader returns, and the error that carries them. -- **JSON Schema** ([`zarr_metadata.json_schema`](https://zarr-metadata.readthedocs.io/en/latest/api/json_schema/)): the - JSON Schema of a `TypedDict`, a v3 declaration, or a whole `zarr.json`. -- **Optional Pydantic integration** ([`zarr_metadata.pydantic`](https://zarr-metadata.readthedocs.io/en/latest/api/pydantic/), - requires Pydantic 2.13 or newer): each model as a Pydantic field type that - validates raw documents through the same strict parser. - -## What this is for - -- **Static Type Checking** The public `TypedDict` definitions describe the static JSON shape of Zarr - metadata. - -- **Dynamic JSON validation** For strict, loc-aware validation of JSON loaded from disk, use the - model parser: - - ```python test="true" session="readme" - import json - from zarr_metadata.model import ZarrV3ArrayMetadata - - with open("zarr.json", "rb") as f: - raw = json.load(f) - - metadata = ZarrV3ArrayMetadata.from_json(raw) - ``` - - The optional Pydantic integration delegates raw input to the same strict - parser and returns the same normalized model class: - - ```python test="true" session="readme" - from pydantic import TypeAdapter - import zarr_metadata.pydantic as zmp - - adapter: TypeAdapter[zmp.ZarrV3ArrayMetadataField] = TypeAdapter(zmp.ZarrV3ArrayMetadataField) - metadata = adapter.validate_python(raw) - encoded = metadata.to_key_value()["zarr.json"] - ``` - - A field annotated `Annotated[ZarrV3ArrayMetadata, zmp.InScope(CONTEXT)]` - is read in `CONTEXT`, whatever the validation context holds, and a model - instance given to it is read again in `CONTEXT`. - - Note that a bare `TypeAdapter` over a public `TypedDict` is a **coercive** shape - adapter, not a Zarr conformance validator; it may coerce values or discard - members that the strict model parser rejects. This is better suited for handling input provided - from Python objects (e.g., user-authored dicts) than JSON. - -## Validation boundary - -The readers and model constructors enforce the declared document structure and a small set -of context-free consistency rules, e.g.: -- fixed format literals, -- finite JSON numbers outside `attributes` and the members nothing in scope reads -- non-negative dimensions -- one `dimension_names` entry per array dimension. - -In a v3 document, validation also checks each extension point: the data -type, the chunk grid, the chunk key encoding, each codec, and each storage -transformer. Each one is checked by the definition that claims its name in -a scope. The scope is `CORE_AND_EXTENSIONS_CONTEXT` unless a `context` is passed. -`ZARR_PYTHON_CONTEXT` adds the data types that zarr-python writes but no -specification registers. -If the definition rejects a configuration, the reader rejects it too, -and a configuration key the definition does not declare is reported as -`unknown_key`. If no definition in the scope claims a name, the reader -does not check that extension point, and the consumer decides whether to -support it. - -Some checks relate one part of a v3 document to another: - -- The fill value is checked against the data type the document names, by - that data type's definition. -- The chunk grid is checked against the shape, by the grid's definition. -- The codecs are checked as a pipeline. In order, each codec's definition - checks the codec against the chunk it receives. This includes a shard's - inner codecs and index codecs. - -The readers do no arithmetic on values. For example, they do not check -whether a fill value survives a `cast_value` round trip. - -Some choices that the spec text leaves open, or answers in two ways: - -- **`attributes` may hold `NaN`, `Infinity` and `-Infinity`.** The spec - does not interpret attributes, and zarr-python and xarray write these - numbers there (for example, a CF `_FillValue`). The models read them, - and `to_key_value` writes them back as bare tokens, which a strict JSON - parser rejects. So may a member the spec does not define when no - definition in scope claims it: nothing reads it, so it is kept as - written. `check`, from `zarr_metadata.typed_json`, rejects a non-finite - number anywhere, including in attributes. -- **A reader walks at most 256 levels of nesting.** A value nested deeper - is an `invalid_value` problem at the first level past the limit, - wherever it is. Every reader, writer and comparison uses one stack - frame per level, and `copy.deepcopy` (and `pickle` before Python 3.12) - uses two. A document at the limit therefore uses about half of the - interpreter's default recursion limit, and the rest is left for the - caller. -- **`consolidated_metadata: null` is a problem.** A zarr-python 3.0.x bug - wrote it. The spec requires an object, and the package accepts only an - object. A reader of those stores removes the key before reading. -- **An extension name must follow the spec's naming rule**, - `^[a-z][a-z0-9-_.]+$`, or be a URI, which earlier versions of the spec - required. Any other name is rejected before any definition is - consulted, so `""` and `"foo/bar"` are problems, not unknown extensions. -- **`must_understand: false` is rejected at every extension point**, - including codecs and storage transformers, although the core spec names - only the data type, chunk grid and chunk key encoding. A reader that - skips a codec reads wrong bytes, just as a reader that skips a data type - reads wrong values. On an unknown top-level member, `must_understand` - keeps its meaning: a reader can skip that member. -- **A v2 datetime or timedelta dtype must have a unit**, as in ` Iterator[ValidationProblem]: + if "dictionary" in configuration and "dictionary_size" not in configuration: + yield ValidationProblem( + ("dictionary_size",), "a dictionary needs its size", "missing_key" + ) + + +ACME_LZ4 = CodecDefinition( + name="acme.lz4", + configuration=AcmeLz4Configuration, + kind="bytes_bytes", + size="dynamic", + rules=acme_lz4_rules, +) +CONTEXT = CORE_AND_EXTENSIONS_CONTEXT.union(ZarrV3Context(codecs=[ACME_LZ4])) +``` + +Pass `context=CONTEXT` to any reader or model constructor. A document that +names `acme.lz4` is then checked against this definition. + +## Rules for the pieces + +**The `TypedDict`** + +- It must be a `typing_extensions.TypedDict`. +- It must say how undeclared keys are handled: `closed=True`, + `extra_items=`, or `closed=False`. An open-by-default `TypedDict` is + refused, because it would accept a misspelled key silently. + +**Its members** must be JSON shapes: + +- `int`, `float`, `bool`, `str`, `None` or `JSONValue`, +- a `Literal`, +- `tuple[T, ...]` or `tuple[T1, T2]`, +- a union, +- a `TypedDict` or `Mapping[str, V]`, +- a `NewType` or a type alias. + +A number may carry bounds from annotated-types: `Gt`, `Ge`, `Lt`, `Le` or +`Interval`. Any other `Annotated` metadata is refused when the definition +is built. + +**The rules function** + +- It runs only on a configuration within its bounds, so it can rely on + them. +- It reports the location of each problem. The value there becomes the + problem's `input`. +- Show values in messages with `zarr_metadata.problems.shown`. Then they + read like the package's own messages. + +**Pickling.** Define each function at module top level. A model holds the +definitions that read it, so the model can be pickled only if they can. + +## Other kinds of definition + +Other kinds declare more: + +- A data type declares its fill value (`fill_value`, `fill_value_rules`, + `fill_value_canonical`). It also declares how its values are stored + (`storage`). +- A chunk grid declares which shapes it fits (`shape_rules`) and its chunk + lengths (`chunk_lengths`). +- A codec declares: + - what it checks about the chunk it receives (`chunk_rules`), + - what it hands on (`transition`), + - any pipelines it contains (`pipelines`). + +The [`zarr_metadata.v3.definition`](api/v3/definition.md) docstring +describes each of these. diff --git a/packages/zarr-metadata/docs/index.md b/packages/zarr-metadata/docs/index.md index b139dfd307..da545079e3 100644 --- a/packages/zarr-metadata/docs/index.md +++ b/packages/zarr-metadata/docs/index.md @@ -1,411 +1,92 @@ # zarr-metadata -Basic tools for modelling Zarr metadata, with minimal dependencies. +Types, models and validators for Zarr metadata documents, with minimal +dependencies. `zarr-metadata` is developed in the [zarr-python repository](https://github.com/zarr-developers/zarr-python/tree/main/packages/zarr-metadata) -and released independently of `zarr` itself. Install it with: +and released independently of `zarr`. Install it with: ``` pip install zarr-metadata ``` -## Who needs this - -This library supports software that interacts with Zarr metadata documents. - ## What this is -This is *not* a full Zarr implementation. It's a collection of data structures and routines that -closely model the content of the Zarr specifications, such as: - -- **Typed JSON shapes** ([`zarr_metadata.v2`](api/v2.md) and - [`zarr_metadata.v3`](api/v3/index.md)): `TypedDict` definitions and - `Literal` aliases for the JSON documents specified by the - [Zarr v2](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html) and - [Zarr v3](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html) - specifications, plus types for - [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/) and a - few widely-used-but-unspecified entities (e.g. consolidated metadata). -- **Document models** ([`zarr_metadata.model`](api/model.md)): immutable - models of whole metadata documents, each one a document and the scope - it was read in, with readers that report every problem at its - location, and store-key (de)serialization. A document - produced by `to_json` shares no mutable state with the model that produced - it. -- **Definitions and scopes** ([`zarr_metadata.definition`](api/definition.md), - with each format's kinds in `zarr_metadata.v3.definition` and - `zarr_metadata.v2.definition`): each extension's configuration as a - `TypedDict` with its rules, and the scope that decides which definitions - read a document. `zarr_metadata.v3` and `zarr_metadata.v2` export every - definition and scope of their format, such as `BYTES_CODEC` and - `CORE_AND_EXTENSIONS_CONTEXT`. -- **JSON type checking** ([`zarr_metadata.typed_json`](api/typed_json.md)): - `check` checks a JSON value against any `TypedDict` and reports every - problem with its location. -- **Problems** ([`zarr_metadata.problems`](api/problems.md)): the problem - values every reader returns, and the error that carries them. -- **JSON Schema** ([`zarr_metadata.json_schema`](api/json_schema.md)): the - JSON Schema of a `TypedDict`, a v3 declaration, or a whole `zarr.json`. -- **Optional Pydantic integration** ([`zarr_metadata.pydantic`](api/pydantic.md), - requires Pydantic 2.13 or newer): each model as a Pydantic field type that - validates raw documents through the same strict parser. - -## What this is for - -- **Static Type Checking** The public `TypedDict` definitions describe the static JSON shape of Zarr - metadata. - -- **Dynamic JSON validation** For strict, location-aware validation of JSON loaded from disk, use the - model parser: - - ```python test="true" session="index" - import json - from zarr_metadata.model import ZarrV3ArrayMetadata - - with open("zarr.json", "rb") as f: - raw = json.load(f) - - metadata = ZarrV3ArrayMetadata.from_json(raw) - ``` - - The optional Pydantic integration delegates raw input to the same strict - parser and returns the same normalized model class: - - ```python test="true" session="index" - from pydantic import TypeAdapter - import zarr_metadata.pydantic as zmp - - adapter: TypeAdapter[zmp.ZarrV3ArrayMetadataField] = TypeAdapter(zmp.ZarrV3ArrayMetadataField) - metadata = adapter.validate_python(raw) - encoded = metadata.to_key_value()["zarr.json"] - ``` - - A field annotated `Annotated[ZarrV3ArrayMetadata, zmp.InScope(CONTEXT)]` - is read in `CONTEXT`, whatever the validation context holds, and a model - instance given to it is read again in `CONTEXT`. - - Note that a bare `TypeAdapter` over a public `TypedDict` is a **coercive** shape - adapter, not a Zarr conformance validator; it may coerce values or discard - members that the strict model parser rejects. This is better suited for handling input provided - from Python objects (e.g., user-authored dicts) than JSON. - -## Validation boundary - -The readers and model constructors enforce the declared document structure and a small set -of context-free consistency rules, e.g.: -- fixed format literals, -- finite JSON numbers outside `attributes` and the members nothing in scope reads -- non-negative dimensions -- one `dimension_names` entry per array dimension. - -In a v3 document, validation also checks each extension point: the data -type, the chunk grid, the chunk key encoding, each codec, and each storage -transformer. Each one is checked by the definition that claims its name in -a scope. The scope is `CORE_AND_EXTENSIONS_CONTEXT` unless a `context` is passed. -`ZARR_PYTHON_CONTEXT` adds the data types that zarr-python writes but no -specification registers. -If the definition rejects a configuration, the reader rejects it too, -and a configuration key the definition does not declare is reported as -`unknown_key`. If no definition in the scope claims a name, the reader -does not check that extension point, and the consumer decides whether to -support it. - -Making the context and explicit part of metadata parsing allows different libraries, which might support different -sets of Zarr features, to formally represent exactly the codecs, data types, etc they support when -parsing metadata. - -Some checks relate one part of a v3 document to another: - -- The fill value is checked against the data type the document names, by - that data type's definition. -- The chunk grid is checked against the shape, by the grid's definition. -- The codecs are checked as a pipeline. In order, each codec's definition - checks the codec against the chunk it receives. This includes a shard's - inner codecs and index codecs. - -The readers do no arithmetic on values. For example, they do not check -whether a fill value survives a `cast_value` round trip. - -Some choices that the spec text leaves open, or answers in two ways: +A collection of data structures and routines that closely model the +metadata rules in the [Zarr v2](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html) +and [Zarr v3](https://zarr-specs.readthedocs.io/en/latest/v3/core/index.html) +specifications and in [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/): -- **`attributes` may hold `NaN`, `Infinity` and `-Infinity`.** The spec - does not interpret attributes, and zarr-python and xarray write these - numbers there (for example, a CF `_FillValue`). The models read them, - and `to_key_value` writes them back as bare tokens, which a strict JSON - parser rejects. So may a member the spec does not define when no - definition in scope claims it: nothing reads it, so it is kept as - written. `check`, from `zarr_metadata.typed_json`, rejects a non-finite - number anywhere, including in attributes. -- **A reader walks at most 256 levels of nesting.** A value nested deeper - is an `invalid_value` problem at the first level past the limit, - wherever it is. Every reader, writer and comparison uses one stack - frame per level, and `copy.deepcopy` (and `pickle` before Python 3.12) - uses two. A document at the limit therefore uses about half of the - interpreter's default recursion limit, and the rest is left for the - caller. -- **`consolidated_metadata: null` is a problem.** A zarr-python 3.0.x bug - wrote it. The spec requires an object, and the package accepts only an - object. A reader of those stores removes the key before reading. -- **An extension name must follow the spec's naming rule**, - `^[a-z][a-z0-9-_.]+$`, or be a URI, which earlier versions of the spec - required. Any other name is rejected before any definition is - consulted, so `""` and `"foo/bar"` are problems, not unknown extensions. -- **`must_understand: false` is rejected at every extension point**, - including codecs and storage transformers, although the core spec names - only the data type, chunk grid and chunk key encoding. A reader that - skips a codec reads wrong bytes, just as a reader that skips a data type - reads wrong values. On an unknown top-level member, `must_understand` - keeps its meaning: a reader can skip that member. -- **A v2 datetime or timedelta dtype must have a unit**, as in ` configuration["length_bytes"]: + yield ValidationProblem((), "expected fewer bytes than the type holds", "invalid_value") + + +PACKED = DataTypeDefinition( + name="acme.packed", + configuration=PackedConfiguration, + fill_value=Base64Bytes, + fill_value_rules=packed_fill_value_rules, +) +context = CORE_AND_EXTENSIONS_CONTEXT.union(ZarrV3Context(data_types=[PACKED])) +data_type, _ = resolve( + {"name": "acme.packed", "configuration": {"length_bytes": 2}}, DataTypeDefinition, context +) +for fill_value in ("YWI=", "not base64!", "YWJj"): + print(fill_value, [problem.message for problem in fill_value_problems(data_type, fill_value)]) +# YWI= [] +# not base64! ['expected standard-alphabet base64, got "not base64!"'] +# YWJj ['expected fewer bytes than the type holds'] +``` + +A reader of a document in this context checks its fill value with these rules, and so does a pydantic field. A JSON Schema written for the context does not contain the rules, so it accepts any string here. -[^1]: The one exception is the `r*` data type, which is identified by a templated string with no `configuration` object. This mixes the identifier with the configuration in the same JSON value, which is not good design. So this library normalizes `r*` into the same form as all the other metadata objects: the declaration `"r16"` keeps its name, and is read by the `r*` definition with the configuration `{"bits": 16}`. +[^1]: The one exception is the [`r*` data type](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#data-types), which is identified by a templated string with no `configuration` object. This mixes the identifier with the configuration in the same JSON value, which is not good design. So this library normalizes `r*` into the same form as all the other metadata objects. The declaration `"r16"` keeps its name. The `r*` definition reads it with the configuration `{"bits": 16}`. diff --git a/packages/zarr-metadata/docs/pydantic.md b/packages/zarr-metadata/docs/pydantic.md new file mode 100644 index 0000000000..5316c00534 --- /dev/null +++ b/packages/zarr-metadata/docs/pydantic.md @@ -0,0 +1,66 @@ +# Pydantic + +`zarr_metadata.pydantic` is an optional integration. + +- It requires Pydantic 2.13 or newer. +- The core package does not depend on Pydantic. + +Each exported name is a Pydantic field type over one of the models. A raw +document given to the field goes through the same strict reader as +`from_json`. The value of the field is an instance of the model class +itself. + +```python test="true" session="pydantic" +import json +from pydantic import BaseModel, TypeAdapter +import zarr_metadata.pydantic as zmp + +with open("zarr.json", "rb") as f: + raw = json.load(f) + +adapter: TypeAdapter[zmp.ZarrV3ArrayMetadataField] = TypeAdapter(zmp.ZarrV3ArrayMetadataField) +metadata = adapter.validate_python(raw) +encoded = metadata.to_key_value()["zarr.json"] + + +class ArrayManifest(BaseModel): + path: str + metadata: zmp.ZarrV3ArrayMetadataField + + +manifest = ArrayManifest.model_validate({"path": "a/b", "metadata": raw}) +``` + +A static type checker sees `manifest.metadata` as a +`zarr_metadata.model.ZarrV3ArrayMetadata`. + +A failed read raises a Pydantic `ValidationError`. It has one line error +per problem, in the format Pydantic uses for its own errors. + +## Choosing the context + +A field type reads its document in the context given by Pydantic's +validation context: + +- If the validation context is a `Context`, every field type of that + format reads in it. +- If it is a mapping: + - the v3 field types read in its `"zarr_metadata_context"` item; + - the v2 field types read in its `"zarr_metadata_context_v2"` item. +- If no context is given for a format, its field types read in that + format's default: `CORE_AND_EXTENSIONS_CONTEXT` or `CORE_V2_CONTEXT`. + +To pin a field to one context, annotate it with `InScope`. A field +annotated `Annotated[ZarrV3ArrayMetadata, zmp.InScope(CONTEXT)]` is read in +`CONTEXT`, whatever the validation context holds. A model instance given to +it is read again in `CONTEXT`. + +## A bare `TypeAdapter` is not a validator + +A `TypeAdapter` over one of the public `TypedDict`s is a **coercive** shape +adapter. It is not a Zarr conformance validator. It may coerce values, or +discard fields that the strict reader rejects. + +- It suits input built from Python objects, such as user-authored dicts. +- For JSON read from a store, use the field types above, or the readers + directly. diff --git a/packages/zarr-metadata/docs/repair.md b/packages/zarr-metadata/docs/repair.md new file mode 100644 index 0000000000..f4f197e40e --- /dev/null +++ b/packages/zarr-metadata/docs/repair.md @@ -0,0 +1,59 @@ +# Repairing known writer bugs + +The readers are strict. If a known writer bug made a document invalid, the +readers report problems. + +The repair readers in `zarr_metadata.model` handle these documents: + +- `read_repaired_node_metadata_v3` +- `read_repaired_node_metadata_v2` +- `read_repaired_consolidated_metadata_v2` + +Each one works in three steps: + +1. Undo the known bugs. +2. Read the result strictly. +3. Return each change it made as a `Repair`. + +## Bugs the repair readers undo + +- **A chunk length of 0 along an empty dimension**, in a v3 + [`regular` chunk grid](https://zarr-specs.readthedocs.io/en/latest/v3/chunk-grids/regular-grid/index.html#description). + zarr-python 3.0 and 3.1 wrote it. The repair writes 1. +- **`consolidated_metadata: null` on a v3 group.** zarr-python 3.0 and 3.1 + wrote it. The spec + [requires an object](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L802-L816). + The repair removes the key. +- **A `consolidated_metadata` member in a `.zgroup` entry of a + `.zmetadata`**, below the root. zarr-python 3 writes it. The + [v2 spec](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html#groups) + allows no key but `zarr_format` in a `.zgroup`. The repair removes the + member. +- **A bare `NaN`, `Infinity` or `-Infinity` as a v2 fill value.** + zarr-python 3.0.0 to 3.0.6 wrote it. The repair writes the string the + [v2 spec](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html#fill-value-encoding) + uses, such as `"NaN"`. +- **A bare non-finite float or complex fill value in a v3 group's + consolidated metadata.** zarr-python 3.0.0 to 3.0.6 wrote it. The repair + writes the string the + [v3 spec](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/data-types/index.html#permitted-fill-values) + uses, such as `"NaN"`. + +## Bugs the repair readers do not undo + +A document with one of these bugs is refused: + +- **The fill value of an `S` or `V` array as a JSON array of byte values**, + in a `.zmetadata` entry. zarr-python 3.0.7 to 3.0.10 wrote it. The + array's own `.zarray` file holds the + [base64 string](https://zarr-specs.readthedocs.io/en/latest/v2/v2.0.html#fill-value-encoding) + the v2 spec requires, and the package reads that file. +- **The fill value of a v2 datetime array as an ISO 8601 string**, not as a + count of ticks. zarr-python 3.0.7 and 3.0.8 wrote it. +- **A chunk length of 0 along a dimension that is not empty**, in a v3 + `regular` chunk grid. zarr-python 3.0 and 3.1 wrote it for an explicit + chunk length of 0. +- **A chunk length of `true` or `false`** in a v3 `regular` chunk grid. + Some releases of zarr-python from 3.0 to 3.2 wrote `true` for + `chunks=True`. Some releases of 3.0 and 3.1 wrote `false` for + `chunks=False`. diff --git a/packages/zarr-metadata/docs/validation-rules.md b/packages/zarr-metadata/docs/validation-rules.md new file mode 100644 index 0000000000..5a0b194c21 --- /dev/null +++ b/packages/zarr-metadata/docs/validation-rules.md @@ -0,0 +1,148 @@ +# What the readers check + +The readers and model constructors enforce the declared document +structure. They also check a small set of consistency rules that need no +context. For example: + +- fixed format literals + ([`zarr_format`](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#zarr-format) + and + [`node_type`](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#node-type)), +- finite JSON numbers, except in `attributes` and in members that nothing + in the context reads, +- non-negative + [dimensions](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#array-metadata-shape), +- one + [`dimension_names`](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#dimension-names) + entry per array dimension. + +## Extension points + +In a v3 document, the readers also check each +[extension point](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#extension-points): + +- the data type, +- the chunk grid, +- the chunk key encoding, +- each codec, +- each storage transformer. + +The definition that claims a name in the context checks that extension +point. The context is `CORE_AND_EXTENSIONS_CONTEXT` unless you pass a +`context`. + +- If the definition rejects a configuration, the reader rejects it too. +- A configuration key the definition does not declare is reported as + `unknown_key`. +- If no definition claims a name, the reader does not check that extension + point. The consumer decides whether to support it. + +The [Context](context.md) page shows this with an example. + +## Checks across the document + +Some checks relate one part of a v3 document to another: + +- The + [fill value](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#fill-value) + is checked against the data type the document names. The data type's + definition does this check. +- The + [chunk grid](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#chunk-grids) + is checked against the shape. The grid's definition does this check. +- The codecs are checked as a + [pipeline](https://github.com/zarr-developers/zarr-specs/blob/v3.1.0/docs/v3/core/index.rst#L931-L1024). + In order, each codec's definition checks the codec against the chunk it + receives. This includes a + [shard's](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/codecs/sharding-indexed/index.html#configuration-parameters) + inner codecs and index codecs. + +The readers do no arithmetic on values. For example, they do not check +whether a fill value survives a `cast_value` round trip. + +## Decisions the spec leaves open + +The spec leaves some choices open, or answers them in two ways. Here is +what the package does. + +**`attributes` may hold `NaN`, `Infinity` and `-Infinity`.** + +- The spec + [does not interpret attributes](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#attributes). +- zarr-python and xarray write these numbers there, for example as a CF + `_FillValue`. +- The models read them. `to_key_value` writes them back as bare tokens, + which a strict JSON parser rejects. +- The same goes for a member the spec does not define, when no definition + in the context claims it. Nothing reads it, so it is kept as written. +- `check`, from `zarr_metadata.typed_json`, rejects a non-finite number + anywhere, including in attributes. + +**A reader walks at most 256 levels of nesting.** + +- A value nested deeper is an `invalid_value` problem at the first level + past the limit, wherever it is. +- Every reader, writer and comparison uses one stack frame per level. + `copy.deepcopy` uses two, and so does `pickle` before Python 3.12. +- So a document at the limit uses about half of the interpreter's default + recursion limit. The rest is left for the caller. + +**`consolidated_metadata: null` is a problem.** + +- A zarr-python 3.0.x bug wrote it. +- The spec + [requires an object](https://github.com/zarr-developers/zarr-specs/blob/fc7dd9c9beb5a50b87f9b08b00bf50fc0048482f/docs/v3/core/index.rst#L802-L816), + and the package accepts only an object. +- The [repair readers](repair.md) remove the key before reading. + +**An extension name must follow the spec's +[naming rule](https://github.com/zarr-developers/zarr-specs/blob/v3.1.0/docs/v3/core/index.rst#L1571-L1607)**, +`^[a-z][a-z0-9-_.]+$`, or be a URI. + +- Earlier versions of the spec required URIs. +- Any other name is rejected before any definition is consulted. +- So `""` and `"foo/bar"` are problems, not unknown extensions. + +**`must_understand: false` is rejected at every extension point**, +including codecs and storage transformers. + +- The + [core spec](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#must-understand) + names only the data type, chunk grid and chunk key encoding. +- But a reader that skips a codec reads wrong bytes. That is just as bad as + a reader that skips a data type and reads wrong values. +- On an unknown top-level member, `must_understand` keeps its meaning: a + reader can skip that member. + +**A v3 `regular` chunk grid's chunk lengths must be at least 1**, even +along a dimension of length 0. + +- The + [regular grid spec](https://zarr-specs.readthedocs.io/en/latest/v3/chunk-grids/regular-grid/index.html#description) + says "Chunk sizes must be greater than zero". + [zarr-specs#379](https://github.com/zarr-developers/zarr-specs/pull/379) + added that sentence after v3.1.0. +- The + [core spec's wording](https://zarr-specs.readthedocs.io/en/v3.1.0/v3/core/index.html#concepts-and-terminology) + requires less: "non-zero when the corresponding dimensions of the arrays + have non-zero length". But it does not explicitly allow anything more. +- So the package rejects a chunk length of 0. zarr-python 3.0 and 3.1 wrote + one for an empty dimension. + +**A v2 datetime or timedelta dtype must have a unit**, as in ` zarr_metadata.definition': api/definition.md diff --git a/packages/zarr-metadata/src/zarr_metadata/_json.py b/packages/zarr-metadata/src/zarr_metadata/_json.py index 69829d32f4..d5de1537a2 100644 --- a/packages/zarr-metadata/src/zarr_metadata/_json.py +++ b/packages/zarr-metadata/src/zarr_metadata/_json.py @@ -20,7 +20,14 @@ from zarr_metadata._common import JSONValue from zarr_metadata._sentinel import UNSET -ProblemKind = Literal["missing_key", "invalid_type", "invalid_value", "invalid_json", "unknown_key"] +ProblemKind = Literal[ + "missing_key", + "invalid_type", + "invalid_value", + "invalid_json", + "unknown_key", + "unclaimed_declaration", +] """Machine-readable classification of a `ValidationProblem`. - `missing_key`: a required key (document key or store key) is absent. @@ -35,6 +42,10 @@ open since 2023), and many readers reject such a key. It has its own kind so that a caller who tolerates it can distinguish it from a wrong value, and so that it never hides other problems in the same object. +- `unclaimed_declaration`: a declaration that nothing in scope claims, which + a reader must understand to open the node. Only + `read_openable_node_metadata_v3` reports it: the other readers accept an + unclaimed declaration, since that is what keeps the format open. """ diff --git a/packages/zarr-metadata/src/zarr_metadata/definition/__init__.py b/packages/zarr-metadata/src/zarr_metadata/definition/__init__.py index 019725679a..2ba4055ba6 100644 --- a/packages/zarr-metadata/src/zarr_metadata/definition/__init__.py +++ b/packages/zarr-metadata/src/zarr_metadata/definition/__init__.py @@ -116,40 +116,42 @@ a closure cannot be pickled, and a `functools.partial` can be pickled but does not compare equal to its loaded copy. - from collections.abc import Iterator - from typing import Annotated, NotRequired +```python +from collections.abc import Iterator +from typing import Annotated, NotRequired - from annotated_types import Ge - from typing_extensions import TypedDict +from annotated_types import Ge +from typing_extensions import TypedDict - from zarr_metadata import ValidationProblem - from zarr_metadata.definition import Nested - from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition, ZarrV3Context +from zarr_metadata import ValidationProblem +from zarr_metadata.definition import Nested +from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition, ZarrV3Context - class AcmeLz4Configuration(TypedDict, closed=True): - acceleration: Annotated[int, Ge(1)] - dictionary: NotRequired[str] - dictionary_size: NotRequired[Annotated[int, Ge(1)]] +class AcmeLz4Configuration(TypedDict, closed=True): + acceleration: Annotated[int, Ge(1)] + dictionary: NotRequired[str] + dictionary_size: NotRequired[Annotated[int, Ge(1)]] - def acme_lz4_rules( - configuration: AcmeLz4Configuration, nested: Nested - ) -> Iterator[ValidationProblem]: - if "dictionary" in configuration and "dictionary_size" not in configuration: - yield ValidationProblem( - ("dictionary_size",), "a dictionary needs its size", "missing_key" - ) +def acme_lz4_rules( + configuration: AcmeLz4Configuration, nested: Nested +) -> Iterator[ValidationProblem]: + if "dictionary" in configuration and "dictionary_size" not in configuration: + yield ValidationProblem( + ("dictionary_size",), "a dictionary needs its size", "missing_key" + ) - ACME_LZ4 = CodecDefinition( - name="acme.lz4", - configuration=AcmeLz4Configuration, - kind="bytes_bytes", - size="dynamic", - rules=acme_lz4_rules, - ) - CONTEXT = CORE_AND_EXTENSIONS_CONTEXT.union(ZarrV3Context(codecs=[ACME_LZ4])) +ACME_LZ4 = CodecDefinition( + name="acme.lz4", + configuration=AcmeLz4Configuration, + kind="bytes_bytes", + size="dynamic", + rules=acme_lz4_rules, +) +CONTEXT = CORE_AND_EXTENSIONS_CONTEXT.union(ZarrV3Context(codecs=[ACME_LZ4])) +``` The TypedDict declares how keys it does not declare are handled. With `closed=True`, such a key is a problem, as above. With `extra_items=`, diff --git a/packages/zarr-metadata/src/zarr_metadata/json_schema.py b/packages/zarr-metadata/src/zarr_metadata/json_schema.py index 2296b78652..7ff4bde29d 100644 --- a/packages/zarr-metadata/src/zarr_metadata/json_schema.py +++ b/packages/zarr-metadata/src/zarr_metadata/json_schema.py @@ -15,13 +15,15 @@ accepts a number with no fractional part, such as `1.0`, as an integer, while `check` requires `1`: - from zarr_metadata.json_schema import json_schema - - json_schema(GzipCodecConfiguration) - # {'$schema': 'https://json-schema.org/draft/2020-12/schema', - # 'type': 'object', - # 'properties': {'level': {'type': 'integer', 'minimum': 0, 'maximum': 9}}, - # 'required': ['level'], 'additionalProperties': False} +```python +from zarr_metadata.json_schema import json_schema + +json_schema(GzipCodecConfiguration) +# {'$schema': 'https://json-schema.org/draft/2020-12/schema', +# 'type': 'object', +# 'properties': {'level': {'type': 'integer', 'minimum': 0, 'maximum': 9}}, +# 'required': ['level'], 'additionalProperties': False} +``` **One v3 declaration.** `declaration_json_schema_v3(kind, context=None)` writes one declaration read as `kind` in the scope `context`: a declaration that a definition in diff --git a/packages/zarr-metadata/src/zarr_metadata/model/__init__.py b/packages/zarr-metadata/src/zarr_metadata/model/__init__.py index 0228861744..2501347273 100644 --- a/packages/zarr-metadata/src/zarr_metadata/model/__init__.py +++ b/packages/zarr-metadata/src/zarr_metadata/model/__init__.py @@ -25,7 +25,12 @@ `read_repaired_node_metadata_v3` or `read_repaired_node_metadata_v2`. They undo each bug with `repair_node_metadata_v3` or `repair_node_metadata_v2` before the strict read, and report what they -changed. `zarr_metadata.json_schema` writes a JSON Schema of what the v3 +changed. A reader accepts a declaration that nothing in its scope claims +(an unknown codec, or an unknown top-level member), and a model lists each +one a reader must understand in `unclaimed_declarations`. To also report +each one as a problem, so that a reading has no problems exactly when a +reader can open the node, use `read_openable_node_metadata_v3`. +`zarr_metadata.json_schema` writes a JSON Schema of what the v3 readers read, except for the rules. One declaration on its own is read with `zarr_metadata.definition.resolve`. @@ -91,6 +96,7 @@ read_consolidated_metadata_v3, read_group_metadata_v3, read_node_metadata_v3, + read_openable_node_metadata_v3, ) from zarr_metadata.model._repair import ( Repair, @@ -160,6 +166,7 @@ "read_group_metadata_v3", "read_node_metadata_v2", "read_node_metadata_v3", + "read_openable_node_metadata_v3", "read_repaired_consolidated_metadata_v2", "read_repaired_node_metadata_v2", "read_repaired_node_metadata_v3", diff --git a/packages/zarr-metadata/src/zarr_metadata/model/_array_v3.py b/packages/zarr-metadata/src/zarr_metadata/model/_array_v3.py index 061819fb3d..fa175ed005 100644 --- a/packages/zarr-metadata/src/zarr_metadata/model/_array_v3.py +++ b/packages/zarr-metadata/src/zarr_metadata/model/_array_v3.py @@ -443,14 +443,14 @@ def must_understand_subset( } -def waived_members(values: Mapping[str, ZarrV3ExtensionField]) -> list[str]: +def waived_members(values: Mapping[str, object]) -> list[str]: """The keys of the extra fields in `values` (by key, as written) that say `"must_understand": false`, which a reader may ignore.""" return [key for key, value in values.items() if _waived(value)] -def _waived(value: ZarrV3ExtensionField) -> bool: +def _waived(value: object) -> bool: """Whether `value`, an extra field as written, is an object that says `"must_understand": false`.""" - return isinstance(value, Mapping) and value.get("must_understand") is False + return is_object(value) and value.get("must_understand") is False class ZarrV3ArrayMetadataUpdate(TypedDict, total=False, closed=True): @@ -835,6 +835,10 @@ def from_json( Raises `MetadataValidationError` with every problem the read finds. `read_array_metadata_v3` returns the reading this model is built from, and the problems of a document that has any. + A declaration that nothing in `context` claims, such as an unknown + codec or an unknown top-level member, is not a problem; the model + lists each one a reader must understand in `unclaimed_declarations`. + `read_openable_node_metadata_v3` reports each one as a problem. """ return cls(data, context=context) @@ -919,7 +923,9 @@ def read_array_metadata_v3( type. Unknown top-level keys are allowed (they map to `extra_fields`). A reader must understand each one that does not declare `must_understand: false`; the model reports these as - `must_understand_fields`. + `must_understand_fields`, and every declaration a reader must + understand, at any depth, as `unclaimed_declarations`. + `read_openable_node_metadata_v3` reports each of those as a problem. To apply a policy over the declarations (for example, allow only the core spec's declarations), iterate over the reading's `declarations()`. A value that diff --git a/packages/zarr-metadata/src/zarr_metadata/model/_group_v3.py b/packages/zarr-metadata/src/zarr_metadata/model/_group_v3.py index 42f3e61357..82aad927ab 100644 --- a/packages/zarr-metadata/src/zarr_metadata/model/_group_v3.py +++ b/packages/zarr-metadata/src/zarr_metadata/model/_group_v3.py @@ -450,7 +450,10 @@ def from_json( scope that claims the member, as `CORE_CONTEXT` does, this is a problem, because the spec requires an object. Use `read_repaired_node_metadata_v3` to read such a store. A member the - spec does not define is kept in `extra_fields`. + spec does not define is kept in `extra_fields`, and is not a + problem when nothing in `context` claims it; the model lists each + such member a reader must understand in `unclaimed_declarations`. + `read_openable_node_metadata_v3` reports each one as a problem. """ return cls(data, context=context) @@ -769,6 +772,44 @@ def read_node_metadata_v3( return ZarrV3UnknownNodeMetadataReading(problems) +def read_openable_node_metadata_v3( + value: object, *, context: ZarrV3Context | None = None +) -> ZarrV3NodeMetadataReading: + """Read `value`, a v3 `zarr.json`, in `context` as `read_node_metadata_v3` does, and also report each declaration a reader must understand to open the node that nothing in `context` claims. + + The spec requires a reader to refuse a node with a declaration it does + not understand: an unknown codec, data type or chunk grid, or an + unknown top-level member that does not say `"must_understand": false`. + `read_node_metadata_v3` accepts these, as `UnclaimedDeclaration`s, + since that is what keeps the format open. This reader reports each one + as a problem of kind `unclaimed_declaration`, at its location, and + gives no model when there is one. A member that says + `"must_understand": false` is ignored whole, with every declaration in + it, as `unclaimed_declarations` ignores it. So the reading has no + problems exactly when a reader that understands `context` can open the + node. The documents a group's `consolidated_metadata` lists are not + read, as `read_group_metadata_v3` does not read them. + """ + reading = read_node_metadata_v3(value, context=context) + # A node reading is only of an array or a group when `value` is an object. + if isinstance(reading, ZarrV3UnknownNodeMetadataReading) or not is_object(value): + return reading + waived = waived_members({key: value[key] for key in reading.extra_fields}) + unclaimed = [ + ValidationProblem( + loc, + f"nothing in scope claims the {kind_name(declaration.read_as)} {declaration.name!r}, " + "which a reader must understand to open the node", + "unclaimed_declaration", + ) + for loc, declaration in unclaimed_declarations_of(reading.declarations(), waived).items() + ] + if len(unclaimed) == 0: + return reading + problems = (*reading.problems, *with_input(unclaimed, value)) + return dataclasses.replace(reading, problems=problems, metadata=None) + + ZarrV3NodeMetadata = TypeAliasType("ZarrV3NodeMetadata", ZarrV3ArrayMetadata | ZarrV3GroupMetadata) """The model of a v3 `zarr.json`: an array model or a group model, as its `node_type` declares.""" @@ -875,7 +916,8 @@ def read_group_metadata_v3( Unknown top-level keys are allowed (they map to `extra_fields`). A reader must understand each one that does not declare `must_understand: false`; the model reports these as - `must_understand_fields`. A `consolidated_metadata` member, if + `must_understand_fields`, and `read_openable_node_metadata_v3` reports + each one as a problem. A `consolidated_metadata` member, if present and claimed by the scope, as `CORE_CONTEXT` claims it, is checked against the envelope the core spec fixes. The documents it lists are not read; `read_consolidated_metadata_v3` reads them. This is the diff --git a/packages/zarr-metadata/src/zarr_metadata/pydantic.py b/packages/zarr-metadata/src/zarr_metadata/pydantic.py index 4434660d8f..308519c0ef 100644 --- a/packages/zarr-metadata/src/zarr_metadata/pydantic.py +++ b/packages/zarr-metadata/src/zarr_metadata/pydantic.py @@ -29,9 +29,11 @@ For example: - TypeAdapter(zmp.ZarrV3ArrayMetadataField).validate_python(document, context=CONTEXT) - ArrayManifest.model_validate(data, context={"zarr_metadata_context": CONTEXT}) - Mixed.model_validate(data, context={"zarr_metadata_context": V3, "zarr_metadata_context_v2": V2}) +```python +TypeAdapter(zmp.ZarrV3ArrayMetadataField).validate_python(document, context=CONTEXT) +ArrayManifest.model_validate(data, context={"zarr_metadata_context": CONTEXT}) +Mixed.model_validate(data, context={"zarr_metadata_context": V3, "zarr_metadata_context_v2": V2}) +``` An existing model instance passes through unchanged when no scope is given. When one is, the instance is read again in it with `refined_in`, so @@ -63,11 +65,13 @@ Usage: - import zarr_metadata.pydantic as zmp +```python +import zarr_metadata.pydantic as zmp - class ArrayManifest(BaseModel): - path: str - metadata: zmp.ZarrV3ArrayMetadataField +class ArrayManifest(BaseModel): + path: str + metadata: zmp.ZarrV3ArrayMetadataField +``` Static type checkers see each field type as its core model class, so `manifest.metadata` is a `zarr_metadata.model.ZarrV3ArrayMetadata`. diff --git a/packages/zarr-metadata/src/zarr_metadata/typed_json.py b/packages/zarr-metadata/src/zarr_metadata/typed_json.py index 30ed95f695..982a3d4868 100644 --- a/packages/zarr-metadata/src/zarr_metadata/typed_json.py +++ b/packages/zarr-metadata/src/zarr_metadata/typed_json.py @@ -6,14 +6,16 @@ value against the TypedDict and returns a value of the TypedDict (or None) and every problem, each located in the value: - import json +```python +import json - from zarr_metadata import ZarrV3ArrayMetadataJSON - from zarr_metadata.typed_json import check +from zarr_metadata import ZarrV3ArrayMetadataJSON +from zarr_metadata.typed_json import check - document, problems = check(json.loads(raw), ZarrV3ArrayMetadataJSON) - for problem in problems: - print(problem.loc, problem.kind, problem.message) +document, problems = check(json.loads(raw), ZarrV3ArrayMetadataJSON) +for problem in problems: + print(problem.loc, problem.kind, problem.message) +``` `check` reads a TypedDict as the typing spec defines it, regardless of how its module writes annotations: diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/__init__.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/__init__.py index 528215df03..a1da0f902a 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/__init__.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/__init__.py @@ -37,6 +37,7 @@ BytesDataTypeName, BytesFillValue, base64_bytes, + base64_fill_value_rules, ) from zarr_metadata.v3.data_type.complex64 import ( COMPLEX64_DATA_TYPE, @@ -338,6 +339,7 @@ "ZarrPythonRawBytesDataTypeName", "ZarrPythonRawBytesFillValue", "base64_bytes", + "base64_fill_value_rules", "hex_float16", "hex_float32", "hex_float64", diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/_byte.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/_byte.py index cfc7adddec..f9eda83e16 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/_byte.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/_byte.py @@ -28,6 +28,23 @@ def base64_problems(value: str) -> Iterator[ValidationProblem]: ) +def base64_fill_value_rules( + configuration: object, nested: object, value: object +) -> Iterator[ValidationProblem]: + """Check that a fill value written as a string is standard-alphabet base64: a data type's `fill_value_rules`. + + `Base64Bytes` is a `NewType` of `str`, so a fill value annotated with + it accepts any string. Pass this rule as the data type's + `fill_value_rules` to reject a string that is not base64. A value of + any other shape is not checked here, so the rule also serves a fill + value that may be written another way, such as + `tuple[ByteValue, ...] | Base64Bytes`. A rule that checks more, such + as the number of bytes, yields from this one first. + """ + if isinstance(value, str): + yield from base64_problems(value) + + def base64_decoded(value: str) -> bytes | None: """The bytes that `value` encodes in standard-alphabet base64; None when it is not such base64.""" try: @@ -41,4 +58,10 @@ def base64_spelling(data: bytes) -> str: return base64.b64encode(data).decode("ascii") -__all__ = ["ByteValue", "base64_decoded", "base64_problems", "base64_spelling"] +__all__ = [ + "ByteValue", + "base64_decoded", + "base64_fill_value_rules", + "base64_problems", + "base64_spelling", +] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/bytes.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/bytes.py index b92facd4af..f7e5c572aa 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/bytes.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/bytes.py @@ -5,10 +5,8 @@ """ import base64 -from collections.abc import Iterator from typing import Final, Literal -from zarr_metadata._json import ValidationProblem from zarr_metadata._numpy import Base64Bytes, base64_bytes from zarr_metadata.definition._definition import ( EmptyConfiguration, @@ -18,7 +16,11 @@ DataTypeDefinition, variable_length, ) -from zarr_metadata.v3.data_type._byte import ByteValue, base64_problems, base64_spelling +from zarr_metadata.v3.data_type._byte import ( + ByteValue, + base64_fill_value_rules, + base64_spelling, +) BYTES_DATA_TYPE_NAME: Final = "bytes" """The `data_type` value for the variable-length `bytes` type.""" @@ -34,14 +36,6 @@ """ -def _fill_value_rules( - configuration: EmptyConfiguration, nested: Nested, value: BytesFillValue -) -> Iterator[ValidationProblem]: - """Check that a string fill value is standard-alphabet base64. A fill value written as byte values is not checked here.""" - if isinstance(value, str): - yield from base64_problems(value) - - def _fill_value_canonical( configuration: EmptyConfiguration, nested: Nested, value: BytesFillValue ) -> str: @@ -60,7 +54,7 @@ def _fill_value_canonical( name=BYTES_DATA_TYPE_NAME, configuration=EmptyConfiguration, fill_value=BytesFillValue, - fill_value_rules=_fill_value_rules, + fill_value_rules=base64_fill_value_rules, fill_value_canonical=_fill_value_canonical, storage=variable_length, ) @@ -79,4 +73,5 @@ def _fill_value_canonical( "BytesDataTypeName", "BytesFillValue", "base64_bytes", + "base64_fill_value_rules", ] diff --git a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/variable_length_bytes.py b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/variable_length_bytes.py index 28803e9d80..11c7da2e69 100644 --- a/packages/zarr-metadata/src/zarr_metadata/v3/data_type/variable_length_bytes.py +++ b/packages/zarr-metadata/src/zarr_metadata/v3/data_type/variable_length_bytes.py @@ -10,14 +10,12 @@ """ import base64 -from collections.abc import Iterator from typing import Final, Literal -from zarr_metadata._json import ValidationProblem from zarr_metadata._numpy import Base64Bytes from zarr_metadata.definition._definition import EmptyConfiguration, Nested from zarr_metadata.v3._definition import DataTypeDefinition, variable_length -from zarr_metadata.v3.data_type._byte import base64_problems, base64_spelling +from zarr_metadata.v3.data_type._byte import base64_fill_value_rules, base64_spelling VARIABLE_LENGTH_BYTES_DATA_TYPE_NAME: Final = "variable_length_bytes" """The `data_type` value for the `variable_length_bytes` type.""" @@ -29,13 +27,6 @@ """Permitted JSON shape of the `fill_value` field for `variable_length_bytes`: a `Base64Bytes` string, which is what zarr-python writes and reads.""" -def _fill_value_rules( - configuration: EmptyConfiguration, nested: Nested, value: VariableLengthBytesFillValue -) -> Iterator[ValidationProblem]: - """Check that the fill value is standard-alphabet base64.""" - yield from base64_problems(value) - - def _fill_value_canonical( configuration: EmptyConfiguration, nested: Nested, value: VariableLengthBytesFillValue ) -> str: @@ -47,7 +38,7 @@ def _fill_value_canonical( name=VARIABLE_LENGTH_BYTES_DATA_TYPE_NAME, configuration=EmptyConfiguration, fill_value=VariableLengthBytesFillValue, - fill_value_rules=_fill_value_rules, + fill_value_rules=base64_fill_value_rules, fill_value_canonical=_fill_value_canonical, storage=variable_length, ) diff --git a/packages/zarr-metadata/tests/model/test_unclaimed_declarations.py b/packages/zarr-metadata/tests/model/test_unclaimed_declarations.py index 02b5521882..e66914b6d2 100644 --- a/packages/zarr-metadata/tests/model/test_unclaimed_declarations.py +++ b/packages/zarr-metadata/tests/model/test_unclaimed_declarations.py @@ -7,6 +7,7 @@ import pytest from typing_extensions import TypedDict +from zarr_metadata import UNSET from zarr_metadata.definition import UnclaimedDeclaration from zarr_metadata.model import ( ZarrV2ArrayMetadata, @@ -15,6 +16,8 @@ ZarrV3ArrayMetadata, ZarrV3GroupMetadata, ZarrV3NodeMetadata, + read_node_metadata_v3, + read_openable_node_metadata_v3, ) from zarr_metadata.v3 import CORE_CONTEXT from zarr_metadata.v3.definition import ( @@ -190,3 +193,78 @@ def test_node_type_tells_a_type_checker_which_model_a_node_is() -> None: for node in nodes: if node.zarr_format == 3 and node.node_type == "group": assert node.consolidated() is None + + +@pytest.mark.parametrize( + ("document", "context", "unclaimed", "problems"), + [ + (ARRAY, CORE_CONTEXT, set(), set()), + ({**ARRAY, "codecs": ("bytes", ZSTD)}, CORE_CONTEXT, {("codecs", 1)}, set()), + ( + {**ARRAY, "codecs": (SHARD,)}, + CORE_CONTEXT, + {("codecs", 0, "configuration", "codecs", 1)}, + set(), + ), + ({**ARRAY, "acme": 1}, CORE_CONTEXT, {("acme",)}, set()), + ({**ARRAY, "acme": {"must_understand": False}}, CORE_CONTEXT, set(), set()), + ({**GROUP, "acme": {"name": "acme"}}, CORE_CONTEXT, {("acme",)}, set()), + ({**GROUP, "acme": {"must_understand": False}}, CORE_CONTEXT, set(), set()), + ( + {**ARRAY, "acme.mirror": {"codec": {"name": "acme.zip"}}}, + MIRRORING, + {("acme.mirror", "codec")}, + set(), + ), + ( + {**ARRAY, "acme.mirror": {"codec": {"name": "acme.zip"}, "must_understand": False}}, + MIRRORING, + set(), + set(), + ), + ( + {**ARRAY, "fill_value": "x", "codecs": ("bytes", ZSTD)}, + CORE_CONTEXT, + {("codecs", 1)}, + {("fill_value",)}, + ), + ({**GROUP, "node_type": "acme"}, CORE_CONTEXT, set(), {("node_type",)}), + ("not a node", CORE_CONTEXT, set(), {()}), + ], + ids=[ + "all-claimed", + "a-codec", + "a-codec-inside-a-shard", + "an-extra-member", + "a-waived-extra-member", + "a-group-member", + "a-waived-group-member", + "a-codec-inside-a-claimed-member", + "a-codec-inside-a-claimed-member-that-waives", + "with-other-problems", + "an-unknown-node-type", + "not-an-object", + ], +) +def test_read_openable_node_metadata_v3_reports_what_a_reader_must_understand( + document: object, + context: Any, # noqa: ANN401 - a v3 scope + unclaimed: set[tuple[str | int, ...]], + problems: set[tuple[str | int, ...]], +) -> None: + """`read_openable_node_metadata_v3` keeps every problem `read_node_metadata_v3` finds and adds one of kind `unclaimed_declaration` at the location of each declaration `unclaimed_declarations` lists, with what the document writes there as its input, whether or not the document has other problems. It gives a model exactly when there are no problems, and the model is the one `read_node_metadata_v3` gives.""" + plain = read_node_metadata_v3(document, context=context) + reading = read_openable_node_metadata_v3(document, context=context) + assert reading.problems[: len(plain.problems)] == plain.problems + assert {problem.loc for problem in plain.problems} == problems + added = reading.problems[len(plain.problems) :] + assert {problem.loc for problem in added} == unclaimed + assert all(problem.kind == "unclaimed_declaration" for problem in added) + assert all(problem.input is not UNSET for problem in added) + if plain.metadata is not None: + assert set(plain.metadata.unclaimed_declarations) == unclaimed + if len(reading.problems) == 0: + assert reading.metadata == plain.metadata + assert reading.metadata is not None + else: + assert reading.metadata is None diff --git a/packages/zarr-metadata/tests/test_docs.py b/packages/zarr-metadata/tests/test_docs.py index 7726151126..a05596075d 100644 --- a/packages/zarr-metadata/tests/test_docs.py +++ b/packages/zarr-metadata/tests/test_docs.py @@ -8,12 +8,14 @@ from __future__ import annotations +import textwrap from collections import defaultdict from pathlib import Path import pytest from pytest_examples import CodeExample, EvalExample, find_examples +import zarr_metadata.definition from zarr_metadata.model import ZarrV3ArrayMetadata PACKAGE_ROOT = Path(__file__).parent.parent @@ -71,3 +73,31 @@ def test_every_python_block_says_whether_it_runs() -> None: ) ] assert unmarked == [] + + +def test_the_extensions_page_example_is_the_definition_module_docstring_example() -> None: + """The acme.lz4 example on the extensions page is, line for line, the example in the `zarr_metadata.definition` module docstring. Only the page's copy runs under these tests, so this is what keeps the docstring's copy from drifting.""" + page_blocks = [ + example + for example in _examples() + if example.path == PACKAGE_ROOT / "docs" / "extensions.md" + and example.prefix_settings().get("session") == "extensions" + ] + assert len(page_blocks) == 1, ( + "expected exactly one extensions-session block on docs/extensions.md" + ) + docstring = zarr_metadata.definition.__doc__ + assert docstring is not None, "zarr_metadata.definition has no module docstring" + lines = docstring.splitlines() + marker = "from collections.abc import Iterator" + starts = [i for i, line in enumerate(lines) if line.strip() == marker] + assert len(starts) == 1, f"expected one line {marker!r} in the docstring, found {len(starts)}" + assert lines[starts[0] - 1].strip() == "```python", ( + f"expected {marker!r} to open a ```python fence in the docstring" + ) + end = starts[0] + while end < len(lines) and lines[end].strip() != "```": + end += 1 + assert end < len(lines), "the docstring's ```python fence is not closed" + docstring_example = textwrap.dedent("\n".join(lines[starts[0] : end])).strip() + assert page_blocks[0].source.strip().splitlines() == docstring_example.splitlines() diff --git a/packages/zarr-metadata/tests/v3/data_type/bytes/test_validators.py b/packages/zarr-metadata/tests/v3/data_type/bytes/test_validators.py index 7f41d878f8..c104b81bf4 100644 --- a/packages/zarr-metadata/tests/v3/data_type/bytes/test_validators.py +++ b/packages/zarr-metadata/tests/v3/data_type/bytes/test_validators.py @@ -1,16 +1,16 @@ -"""Cover the `base64_bytes` brand validator. +"""Cover the `base64_bytes` brand validator and the `base64_fill_value_rules` fill value rule. The pydantic-driven fixture tests don't enforce the base64 alphabet or length-multiple-of-4 constraint because `Base64Bytes` is a `NewType`, which pydantic treats as plain `str`. Direct coverage of the validator -function lives here. +function and the rule built on it lives here. """ from __future__ import annotations import pytest -from zarr_metadata.v3.data_type.bytes import base64_bytes +from zarr_metadata.v3.data_type.bytes import base64_bytes, base64_fill_value_rules VALID = [ "", # empty is valid base64 (length 0, multiple of 4) @@ -41,3 +41,16 @@ def test_valid(value: str) -> None: def test_invalid(value: str) -> None: with pytest.raises(ValueError, match="standard-alphabet base64"): base64_bytes(value) + + +# A value written in another shape, such as byte values, is the annotation's to check. +@pytest.mark.parametrize("value", [*VALID, (1, 2, 3), None]) +def test_base64_fill_value_rules_accepts(value: object) -> None: + assert list(base64_fill_value_rules({}, (), value)) == [] + + +@pytest.mark.parametrize("value", INVALID) +def test_error_base64_fill_value_rules_rejects_a_string_that_is_not_base64(value: str) -> None: + assert [(p.loc, p.kind) for p in base64_fill_value_rules({}, (), value)] == [ + ((), "invalid_value") + ]