feat(zarr-metadata): re-export each v3 extension package's submodule names

.github/workflows/test.yml

@@ -201,6 +201,32 @@ jobs:
       run: |
         HATCH_ENV=test.py3.13-minimal just benchmark
 
+  examples:
+    name: Example scripts
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
+      with:
+        fetch-depth: 0 # required for hatch version discovery when uv builds zarr
+        persist-credentials: false
+    - name: Set up Python
+      uses: actions/setup-python@6d4fd22b20465ccc0f7354476588e8df7c9ca0e0 # v7.1.0
+      with:
+        python-version: '3.13'
+        cache: 'pip'
+    - name: Install uv
+      uses: astral-sh/setup-uv@07a047e608db84b3d8c06cd9a7c7dcc9c355c18e # v10.3.2
+    - name: Install Hatch
+      run: python -m pip install hatch==1.16.5
+    - name: Install just
+      uses: extractions/setup-crate@7577c1bdf2d95e6d65d532788f35ed79d4b1dda2 # v2.0.1
+      with:
+        repo: casey/just
+        version: 1.58.0
+    - name: Run Examples
+      run: |
+        HATCH_ENV=test.py3.13-minimal just test-examples
+
   test-complete:
     name: Test complete
 
@@ -209,7 +235,8 @@ jobs:
         test,
         test-upstream-and-min-deps,
         doctests,
-        benchmarks
+        benchmarks,
+        examples
       ]
     if: always()
     runs-on: ubuntu-latest

Justfile

@@ -64,6 +64,10 @@ hypothesis *args:
 doctest *args:
     hatch run doctest:pytest tests/test_docs.py -v "$@"
 
+# Run each example script against the local checkout, in its PEP 723 environment
+test-examples *args:
+    hatch run {{ quote(hatch_env) }}:pytest --run-examples tests/test_examples.py -v "$@"
+
 # Run the benchmark suite
 benchmark *args:
     hatch run {{ quote(hatch_env) }}:pytest --benchmark-enable tests/benchmarks "$@"

changes/4560.bugfix.md

@@ -0,0 +1,8 @@
+Encoding a chunk through a numcodecs filter no longer passes the
+filter's flattened output to the next codec: `_encode_sync` reshapes the
+flattened array to the filter's output shape (from
+`resolve_metadata`), so downstream codecs (e.g. `TransposeCodec` or an `imagecodecs` serializer chained after a
+`Delta` or `FixedScaleOffset` filter) receive a chunk of the expected
+shape instead of a flat 1-D array that could make writing fail or
+silently corrupt data. Shape-changing codecs such as `PackBits` get
+their own resolved output shape.

examples/README.md

@@ -19,7 +19,7 @@ examples/
 To add a new example:
 
 1. Create a new subdirectory: `examples/my_example/`
-2. Add your Python code: `examples/my_example/my_example.py`
+2. Add your Python code: `examples/my_example/my_example.py`. Start it with a [PEP 723](https://peps.python.org/pep-0723/) header that lists its dependencies, including `zarr`. `just test-examples` runs every example against your local checkout, and CI runs it on every pull request.
 3. Create documentation: `examples/my_example/README.md`
 4. Create a documentation page at `docs/user-guide/examples/my_example.md`. The documentation page should simply link to the `README.md` and the source code, e.g.:
 

examples/codec_pipeline_performance/codec_pipeline_performance.py

@@ -1,5 +1,5 @@
 # /// script
-# requires-python = ">=3.11"
+# requires-python = ">=3.12"
 # dependencies = [
 #   "zarr @ git+https://github.com/zarr-developers/zarr-python.git@main",
 #   "numpy",

examples/sharding_coalescing/sharding_coalescing.py

@@ -1,5 +1,5 @@
 # /// script
-# requires-python = ">=3.11"
+# requires-python = ">=3.12"
 # dependencies = [
 #   "zarr @ git+https://github.com/zarr-developers/zarr-python.git@main",
 #   "numpy",

packages/zarr-metadata/README.md

@@ -41,7 +41,7 @@ closely model the content of the Zarr specifications, such as:
   `TypedDict` with its rules, and the scope that decides which definitions
   read a document. `zarr_metadata.v3` and `zarr_metadata.v2` export every
   definition and scope of their format, such as `BYTES_CODEC` and
-  `CORE_AND_EXTENSIONS`.
+  `CORE_AND_EXTENSIONS_CONTEXT`.
 - **JSON type checking** ([`zarr_metadata.typed_json`](https://zarr-metadata.readthedocs.io/en/latest/api/typed_json/)):
   `check` checks a JSON value against any `TypedDict` and reports every
   problem with its location.
@@ -104,7 +104,9 @@ of context-free consistency rules, e.g.:
 In a v3 document, validation also checks each extension point: the data
 type, the chunk grid, the chunk key encoding, each codec, and each storage
 transformer. Each one is checked by the definition that claims its name in
-a scope. The scope is `CORE_AND_EXTENSIONS` unless a `context` is passed.
+a scope. The scope is `CORE_AND_EXTENSIONS_CONTEXT` unless a `context` is passed.
+`ZARR_PYTHON_CONTEXT` adds the data types that zarr-python writes but no
+specification registers.
 If the definition rejects a configuration, the reader rejects it too,
 and a configuration key the definition does not declare is reported as
 `unknown_key`. If no definition in the scope claims a name, the reader
@@ -232,13 +234,13 @@ each field:
 ```python test="true" session="readme"
 from zarr_metadata.model import read_array_metadata_v3
 from zarr_metadata.definition import canonical_of, with_problems
-from zarr_metadata.v3.definition import CORE
+from zarr_metadata.v3.definition import CORE_CONTEXT
 
 reading = read_array_metadata_v3(raw)
 beyond_core = [
     loc
     for loc, field in reading.fields()
-    if field.name is not None and CORE.claimant(field.read_as, field.name) is None
+    if field.name is not None and CORE_CONTEXT.claimant(field.read_as, field.name) is None
 ]
 simplest = {
     loc: canonical_of(field, problems)
@@ -283,15 +285,15 @@ scope claims is an `UnclaimedField` too, also when it sits inside another
 field, such as a codec inside a shard. A model's `unclaimed_fields` lists
 every field nothing in scope claims, at any depth, by its location, and
 leaves out the members that waive `must_understand`: a reader can open
-the node when it is empty. `CORE` holds
+the node when it is empty. `CORE_CONTEXT` holds
 `CONSOLIDATED_METADATA_EXTRA_FIELD`, which reads a group's
 `consolidated_metadata` as the envelope the spec fixes. A group's
 `consolidated()` reads the documents that member lists, in the group's
 own scope.
 
 A v3 model consists of its document and the scope it was read in.
 `ZarrV3ArrayMetadata(document, context=None)` reads the document in the
-scope (`CORE_AND_EXTENSIONS` when none is given) and raises
+scope (`CORE_AND_EXTENSIONS_CONTEXT` when none is given) and raises
 `MetadataValidationError` with every problem, so an invalid model is never
 built. `to_json` returns the document as it was written, and
 `to_key_value` writes that same document to store bytes.
@@ -322,7 +324,7 @@ all of them with `ZarrV3ConsolidatedMetadata`, or with
 `read_consolidated_metadata_v3`, which returns problems instead of
 raising them. A document with a problem then fails only when it is read.
 Pass `context=group.context` to read the documents in the group's own
-scope; with no `context`, they are read in `CORE_AND_EXTENSIONS`.
+scope; with no `context`, they are read in `CORE_AND_EXTENSIONS_CONTEXT`.
 
 To write consolidated metadata, build a `ZarrV3ConsolidatedMetadata` and
 pass it to the group's `update`. The group reads the listed models in its
@@ -332,7 +334,7 @@ models as entries. Such an entry is accepted when its claims refine into
 the model's scope, and refused at its path otherwise. To consolidate
 children that were read in several scopes, consolidate them in the
 `union` of those scopes. Every reader takes `context=None` for the
-default scope, `CORE_AND_EXTENSIONS`.
+default scope, `CORE_AND_EXTENSIONS_CONTEXT`.
 
 Two models are equal when they represent the same document, even if each
 is spelled differently. Values the package interprets (each field, and the

packages/zarr-metadata/changes/4575.feature.md

@@ -0,0 +1 @@
+Added `zarr_metadata.testing.strategies`, Hypothesis strategies that draw valid Zarr v3 metadata fields, behind a new `testing` extra (`pip install zarr-metadata[testing]`). There is one strategy for every codec, data type, chunk grid and chunk key encoding the package defines that holds no other field. `fields` draws every envelope spelling the spec allows, or only the canonical one with `spellings="canonical"`. Codecs can be drawn for the `Chunk` they are handed, chunk grids for an array shape, and fill values for a data type. The strategies are a mapping keyed by definition, so an extension author can add strategies for their own definitions.

packages/zarr-metadata/changes/4583.feature.md

@@ -0,0 +1 @@
+Add the `ZARR_PYTHON_CONTEXT` scope, which reads what zarr-python writes: `CORE_AND_EXTENSIONS_CONTEXT` plus zarr-python's own data types `null_terminated_bytes` (NumPy's `S`), `raw_bytes` (NumPy's `V`) and `variable_length_bytes`, which no specification registers. `CORE_AND_EXTENSIONS_CONTEXT` now also reads the `fixed_length_utf32` data type (NumPy's `U`) and the `vlen-utf8` and `vlen-bytes` codecs that zarr-extensions registers, so the strings zarr-python writes are checked instead of read as unclaimed.

packages/zarr-metadata/changes/4583.removal.md

@@ -0,0 +1 @@
+Rename the scopes `CORE`, `CORE_AND_EXTENSIONS` and `CORE_V2` to `CORE_CONTEXT`, `CORE_AND_EXTENSIONS_CONTEXT` and `CORE_V2_CONTEXT`, so that a scope's name says it is a context. The old names, imported from `zarr_metadata.v3`, `zarr_metadata.v3.definition`, `zarr_metadata.v2` or `zarr_metadata.v2.definition`, still refer to the same scopes with a `DeprecationWarning`, and will be removed in a later release.

packages/zarr-metadata/changes/4588.feature.md

@@ -0,0 +1 @@
+`zarr_metadata.v3.data_type`, `zarr_metadata.v3.codec`, `zarr_metadata.v3.chunk_grid` and `zarr_metadata.v3.chunk_key_encoding` re-export every name their submodules export, so `from zarr_metadata.v3.data_type import BoolDataTypeName, Uint8DataTypeName` works again, as it did before 0.6.0. Each name is the same object as in its submodule, which stays its documented home.

packages/zarr-metadata/docs/api/index.md

@@ -22,7 +22,7 @@ The package is organized to mirror the structure of the Zarr specifications:
 - [`zarr_metadata.v2`](v2.md) — `TypedDict` shapes for Zarr v2 documents
   (`.zarray`, `.zgroup`, `.zattrs`, `.zmetadata`), and
   `zarr_metadata.v2.definition`: the v2 kinds of metadata, the scope
-  `CORE_V2`, and the readers of one `dtype` or codec. Each dtype family's
+  `CORE_V2_CONTEXT`, and the readers of one `dtype` or codec. Each dtype family's
   definition is in `zarr_metadata.v2.data_type`, and each codec's in
   `zarr_metadata.v2.codec`
 - [`zarr_metadata.v3`](v3/index.md) — `TypedDict` shapes for Zarr v3
@@ -35,7 +35,7 @@ The package is organized to mirror the structure of the Zarr specifications:
   scope type `Context`. Its module docstring is the guide
 - [`zarr_metadata.v3.definition`](v3/definition.md) — the five v3 kinds
   of metadata (`CodecDefinition`, `DataTypeDefinition`, ...), the codec
-  pipeline types, and the v3 scopes `CORE` and `CORE_AND_EXTENSIONS`
+  pipeline types, and the v3 scopes `CORE_CONTEXT` and `CORE_AND_EXTENSIONS_CONTEXT`
 
 The document types, models, and spec vocabulary — including the store keys —
 are re-exported at the top level, so

packages/zarr-metadata/docs/api/testing.md

@@ -0,0 +1,8 @@
+---
+title: testing.strategies
+---
+
+::: zarr_metadata.testing.strategies
+    options:
+      inherited_members: false
+      show_if_no_docstring: false

packages/zarr-metadata/docs/api/v3/codec.md

@@ -22,4 +22,8 @@ title: codec
 
 ::: zarr_metadata.v3.codec.transpose
 
+::: zarr_metadata.v3.codec.vlen_bytes
+
+::: zarr_metadata.v3.codec.vlen_utf8
+
 ::: zarr_metadata.v3.codec.zstd

packages/zarr-metadata/docs/api/v3/data_type.md

@@ -40,8 +40,16 @@ title: data_type
 
 ::: zarr_metadata.v3.data_type.string
 
+::: zarr_metadata.v3.data_type.fixed_length_utf32
+
 ::: zarr_metadata.v3.data_type.numpy_datetime64
 
 ::: zarr_metadata.v3.data_type.numpy_timedelta64
 
 ::: zarr_metadata.v3.data_type.struct
+
+::: zarr_metadata.v3.data_type.null_terminated_bytes
+
+::: zarr_metadata.v3.data_type.variable_length_bytes
+
+::: zarr_metadata.v3.data_type.zarr_python_raw_bytes

packages/zarr-metadata/docs/index.md

@@ -39,7 +39,7 @@ closely model the content of the Zarr specifications, such as:
   `TypedDict` with its rules, and the scope that decides which definitions
   read a document. `zarr_metadata.v3` and `zarr_metadata.v2` export every
   definition and scope of their format, such as `BYTES_CODEC` and
-  `CORE_AND_EXTENSIONS`.
+  `CORE_AND_EXTENSIONS_CONTEXT`.
 - **JSON type checking** ([`zarr_metadata.typed_json`](api/typed_json.md)):
   `check` checks a JSON value against any `TypedDict` and reports every
   problem with its location.
@@ -102,7 +102,9 @@ of context-free consistency rules, e.g.:
 In a v3 document, validation also checks each extension point: the data
 type, the chunk grid, the chunk key encoding, each codec, and each storage
 transformer. Each one is checked by the definition that claims its name in
-a scope. The scope is `CORE_AND_EXTENSIONS` unless a `context` is passed.
+a scope. The scope is `CORE_AND_EXTENSIONS_CONTEXT` unless a `context` is passed.
+`ZARR_PYTHON_CONTEXT` adds the data types that zarr-python writes but no
+specification registers.
 If the definition rejects a configuration, the reader rejects it too,
 and a configuration key the definition does not declare is reported as
 `unknown_key`. If no definition in the scope claims a name, the reader
@@ -234,13 +236,13 @@ each field:
 ```python test="true" session="index"
 from zarr_metadata.model import read_array_metadata_v3
 from zarr_metadata.definition import canonical_of, with_problems
-from zarr_metadata.v3.definition import CORE
+from zarr_metadata.v3.definition import CORE_CONTEXT
 
 reading = read_array_metadata_v3(raw)
 beyond_core = [
     loc
     for loc, field in reading.fields()
-    if field.name is not None and CORE.claimant(field.read_as, field.name) is None
+    if field.name is not None and CORE_CONTEXT.claimant(field.read_as, field.name) is None
 ]
 simplest = {
     loc: canonical_of(field, problems)
@@ -285,15 +287,15 @@ scope claims is an `UnclaimedField` too, also when it sits inside another
 field, such as a codec inside a shard. A model's `unclaimed_fields` lists
 every field nothing in scope claims, at any depth, by its location, and
 leaves out the members that waive `must_understand`: a reader can open
-the node when it is empty. `CORE` holds
+the node when it is empty. `CORE_CONTEXT` holds
 `CONSOLIDATED_METADATA_EXTRA_FIELD`, which reads a group's
 `consolidated_metadata` as the envelope the spec fixes. A group's
 `consolidated()` reads the documents that member lists, in the group's
 own scope.
 
 A v3 model consists of its document and the scope it was read in.
 `ZarrV3ArrayMetadata(document, context=None)` reads the document in the
-scope (`CORE_AND_EXTENSIONS` when none is given) and raises
+scope (`CORE_AND_EXTENSIONS_CONTEXT` when none is given) and raises
 `MetadataValidationError` with every problem, so an invalid model is never
 built. `to_json` returns the document as it was written, and
 `to_key_value` writes that same document to store bytes.
@@ -324,7 +326,7 @@ all of them with `ZarrV3ConsolidatedMetadata`, or with
 `read_consolidated_metadata_v3`, which returns problems instead of
 raising them. A document with a problem then fails only when it is read.
 Pass `context=group.context` to read the documents in the group's own
-scope; with no `context`, they are read in `CORE_AND_EXTENSIONS`.
+scope; with no `context`, they are read in `CORE_AND_EXTENSIONS_CONTEXT`.
 
 To write consolidated metadata, build a `ZarrV3ConsolidatedMetadata` and
 pass it to the group's `update`. The group reads the listed models in its
@@ -334,7 +336,7 @@ models as entries. Such an entry is accepted when its claims refine into
 the model's scope, and refused at its path otherwise. To consolidate
 children that were read in several scopes, consolidate them in the
 `union` of those scopes. Every reader takes `context=None` for the
-default scope, `CORE_AND_EXTENSIONS`.
+default scope, `CORE_AND_EXTENSIONS_CONTEXT`.
 
 Two models are equal when they represent the same document, even if each
 is spelled differently. Values the package interprets (each field, and the

packages/zarr-metadata/docs/parsing-validation.md

@@ -161,7 +161,7 @@ This procedure yields one of two values:
 
 `zarr-metadata` does not run each stage as a separate function. Instead, a single read of the document collects the problems from every stage, so a caller gets the full list of problems at once.
 
-The **registry** from the resolve-and-validate stage is called a **scope** in this library. For Zarr V3 it is a [`ZarrV3Context`](api/v3/definition.md) object, with one slot for each kind of field (`data_types`, `codecs`, `chunk_grids`, ...), and every reader takes a `context` argument. The default scope is `CORE_AND_EXTENSIONS`, which holds the data types, codecs, chunk grids, etc. defined in the Zarr V3 spec and in [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/). `CORE` holds only the definitions from the Zarr V3 spec.
+The **registry** from the resolve-and-validate stage is called a **scope** in this library. For Zarr V3 it is a [`ZarrV3Context`](api/v3/definition.md) object, with one slot for each kind of field (`data_types`, `codecs`, `chunk_grids`, ...), and every reader takes a `context` argument. The default scope is `CORE_AND_EXTENSIONS_CONTEXT`, which holds the data types, codecs, chunk grids, etc. defined in the Zarr V3 spec and in [zarr-extensions](https://github.com/zarr-developers/zarr-extensions/). `CORE_CONTEXT` holds only the definitions from the Zarr V3 spec. `ZARR_PYTHON_CONTEXT` adds to `CORE_AND_EXTENSIONS_CONTEXT` the data types that zarr-python writes but no specification registers (`null_terminated_bytes` for NumPy's `S`, `raw_bytes` for NumPy's `V`, and `variable_length_bytes`), so a strict check of what zarr-python writes passes it.
 
 `read_array_metadata_v3` reads a JSON document and returns a "reading". `reading.problems` is a tuple of the problems, and `reading.metadata` is the validated model, or `None` if there are problems. Each problem has a `loc`, which is its position in the document as a tuple, a `kind`, like `missing_key` or `invalid_value`, and a human-readable `message`:
 
@@ -193,10 +193,10 @@ To read one field on its own, such as a codec, use `resolve` from `zarr_metadata
 
 ```python test="true" session="parsing-validation"
 from zarr_metadata.definition import resolve
-from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS, CodecDefinition
+from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition
 
 field, problems = resolve(
-    {"name": "gzip", "configuration": {"level": 12}}, CodecDefinition, CORE_AND_EXTENSIONS
+    {"name": "gzip", "configuration": {"level": 12}}, CodecDefinition, CORE_AND_EXTENSIONS_CONTEXT
 )
 for problem in problems:
     print(problem.loc, problem.kind)
@@ -214,17 +214,17 @@ To make an unknown data type or codec known, add its definition to the scope's s
 
 ```python test="true" session="parsing-validation"
 from zarr_metadata.definition import EmptyConfiguration
-from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS, CodecDefinition
+from zarr_metadata.v3.definition import CORE_AND_EXTENSIONS_CONTEXT, CodecDefinition
 
 # The definition of the codec that was unknown: here, a codec named
 # `acme.codec` that takes no configuration.
 MY_CODEC = CodecDefinition(
     name="acme.codec", configuration=EmptyConfiguration, kind="bytes_bytes", size="dynamic"
 )
-context = CORE_AND_EXTENSIONS.update(codecs=[*CORE_AND_EXTENSIONS.codecs.values(), MY_CODEC])
+context = CORE_AND_EXTENSIONS_CONTEXT.update(codecs=[*CORE_AND_EXTENSIONS_CONTEXT.codecs.values(), MY_CODEC])
 reading = read_array_metadata_v3(raw, context=context)
 ```
 
-`update` replaces each slot it is given and keeps the others. A slot can also be given as a mapping from each definition's name to it, so `CORE_AND_EXTENSIONS.update(codecs={**CORE_AND_EXTENSIONS.codecs, "gzip": MY_GZIP})` replaces the gzip definition with your own.
+`update` replaces each slot it is given and keeps the others. A slot can also be given as a mapping from each definition's name to it, so `CORE_AND_EXTENSIONS_CONTEXT.update(codecs={**CORE_AND_EXTENSIONS_CONTEXT.codecs, "gzip": MY_GZIP})` replaces the gzip definition with your own.
 
 [^1]: The one exception is the `r*` data type, which is identified by a templated string with no `configuration` object. This mixes the identifier with the configuration in the same JSON value, which is not good design. So this library normalizes `r*` into the same form as all the other metadata objects: the field `"r16"` keeps its name, and is read by the `r*` definition with the configuration `{"bits": 16}`.